I don't think it will damage your career at all. If you would be interested in working for another startup in the future, having startup experience would be a huge asset. Not all programmers do well in an environment where they have to do work outside of their specialization or have to accomplish something useful with vague and/or shifting requirements, so other people who run startups might very well leap at the chance to hire someone who likes that kind of work.
You could run into people (if you look for another job as an employee) who will be worried that you'll just ditch them when you have enough money saved up to try another startup, but it's pretty simple to say that you tried the startup thing and discovered it's just not for you. Note that that doesn't have to be strictly true, I don't think employees are any more obligated to disclose all of their potential future plans any more than companies are obligated to give you months of notice that they're going to lay you off. Even people who do plan to stay at a given company for the long term change their minds.
Now, what really could damage your career is if you spend months or years on your startup and end up with nothing to show for it. So, uh, don't do that :) But seriously, as long as you have a reasonable amount of code to show potential employers, you'll be fine. And keep in mind that running a startup involves a huge amount of non-code work, so you don't actually need a huge amount of code.
End of advice about what you actually asked, beginning of advice about starting your own startup:
Please please please don't quit your job immediately. Build a teeny tiny proof of concept (maybe a checklist or a mini-course or a small tool) in your free time and see if anyone is willing to give you money for it. Friends and acquaintances saying your project sounds cool is not enough, you need people who are willing to part with actual money before you even consider quitting your job. It's extremely common for friends and acquaintances to tell you your product is great when they would never actually pay real money for it because they're too nice to say they don't care about the thing you built. If you need more time to work on your product, see if you can negotiate a shorter work week in lieu of a raise or something.
edited to fix a typo