Some of the things you mention seem addressable. E.g.
He has told me twice during training (before I had performed any duties) that others have expressed lot of concerns and nerves around my future performance.
If he says this again, ask him who these people are and suggest that you pair with them. Pairing is when you and another person do something together. In this case, apparently there are some decisions that are risky for you to make alone. So go to the other person and work through the issue with that person. Then, if you are choosing incorrectly, the other person can tell you immediately.
In the near term, pairing like this makes you and the other person less productive. But it also makes you more correct. And of course without you there, the other people would have to carry all of the burden. With you there, they can figure out what they can leave to you and what you need their help to do. Where, over time, the things that you need help to make the right decision should decrease over time.
There is usually a certain amount of legwork in this kind of thing. So the people with whom you pair should be able to quickly determine some things that they can let you do on your own. Do those things first, before you approach the other person. That way, they can spend all their time on helping you understand the decisions you will eventually be making alone. That's the benefit of having you involved. You can take up some of the repetitive work that they had to do. So even if they have to pair, they may still end up doing less work overall. You are now a benefit to those people, not just a danger.
Your goal here is to work towards the point where you can work alone without raising concerns. You should try to do as much of the work as they are comfortable letting you do.
This also shifts things. The people who currently have concerns over what you might do now have incentive to help you do better. Because so long as they view you as too dangerous to work alone, they have to do more work. Currently, they may be able to push this work off on the manager, which may make him cranky. Help him be a better manager. He should delegate things so that he isn't doing everything.
He has complained on a couple of occasions that he hadn't been able to take his break on time because he's been spending time on me.
Does he take breaks at the same time every day? Learn those times and avoid them. If approaching one, deliberately wait until he is returning from the break to start the next conversation.
If his breaks are taken on a more freeform basis, make a habit of asking him if he has time now or if he is due for a break. Then if he complains that he missed a break, remind him that you had asked and suggest that he let you know these things in the future.
Pairing can also help with this. If you always need to go to the manager for help, naturally this is more work for him. But if you are pairing and go to that person for help, you're not putting the same burden on the manager.
Addressing those concerns that you can will make you appear proactive. Because ultimately, you want to convince your manager and other coworkers that having you there is good. Realistically, they should realize that most workers do not add net value for six months or so after being hired. It is unrealistic to expect you to decrease their workload immediately. But you want them to see light at the end of the tunnel.