Although your question title focuses on whether it is rude/ethical to cancel an examination you agreed to, I believe your actual question hinges on this statement:
But then, after much thought, I am not confident enough in taking the
exam and I would like to cancel the scheduled date and tell them my
reasons.
As Jane S. mentioned in the comments, the company offered you the chance to take the examination because they saw potential in your ability. You agreed to take the examination, and now it sounds like you are having doubts about your abilities. This is completely natural. But it doesn't mean you should pull out.
"Should I just take the exam even though I know I'll fail it?"
The answer to this is almost certainly "Yes". Even if you know with 100% certainty you'll fail, it's still in your best interest to take the exam. Think of it this way: what's the worst that will happen if you fail? Answer: you won't get the job. But if you withdraw from the exam without taking it, you certainly won't get the job either. On the other hand, taking this sort of exam and failing still has potential benefits:
Taking an exam like this is valuable practice for general hiring procedures. If you experience this one (and fail it), you will remember this experience for future applicatioins.
This is not a university-style test. Even if you can't answer many questions, you won't know where their "passing mark" is for the position they're hiring for. You could quite possibly "fail" the exam by your own standards, but still be considered good enough for the position they're hiring for. You won't know unless you try it.
If it's an oral test, you will probably get feedback during the exam of how you are doing, which areas you were good at, and which you need to improve on. If you don't end up getting the job, you can use this feedback to your advantage for future job applications/interviews/exams.