I have worked in one kind of job, but realized that it is not the right job for me.I decided to apply for both new kind of jobs and the original type of job, but it seems I am stuck.
Don't despair - you aren't the only person who got into a job/career and found it wasn't for them!
For current type of job, I am not good enough. But on the other side, when I apply for new kind of job, I am rejected, probably because I do not have the right job experience/school.
You're probably on the money there - in general terms having school + experience is your best bet, having experience (but no school) probably second and finally having just the school qualifications.
That's not to say you can't get a role in a new career without any of these - but it's going to be a tough sell unless you're moving from a career where you have some obviously transferable skills, and you'd probably have to start right at the bottom.
If you're applying for roles that explicitly list qualifications (be that academic or experience) that you don't have your chances are going to be extremely, extremely slim. If there's no required qualifications - or there are ones you have or equivalents to then you can apply and make your case in the CV and cover letter as you have been doing.
What should I do now?
If roles in your new desired career are consistently asking for a qualification you don't have then it's time to break out the ol' textbooks and go get that qualification!