By and large, the only opinion about your performance that truly matters for your employment prospects at this or future employer is that of your boss.
Colleagues can think many things. In the end what matters is (a) your ability to support a good performance record with evidence, and (b) your boss's inferences regarding your performance based on that evidence.
Sounds like you may be taking work-related issues personally, and that's a common pitfall -- especially if deep down you believe your performance is good or adequate, and your conscience is clear (which seems to be the case).
A common mistake in such situation is to fall victim to your own insecurity tied to what others think of you, and to engage in proactive 'repair' behavior attempting to solve problems that appear to you to exist. However, if you take a step back, you will often see that these issues are mostly in your head, there is little to no evidence to support the rumors, and that you trying to problem-solve these issues is the only thing that makes them real and serious. Avoid giving colleagues 'ammunition' for their gossip by confronting them about the rumors, because this is what validating the rumors in the first place.
If you do not react, they will simply run out of actionable information on which to base their gossip, and sooner or later it will run out of steam and die down. Time and patience are on your side.
If at any point you will hear specific remarks addressed to you personally which seem inappropriate, insulting, rude, and unsubstantiated, you have the right to bring this to your boss's attention and seek resolution.
Until that happpens, I would recommend to stay focused on your immediate work tasks, and to maintain a cordial professional (but not overly friendly or personal) relationship with your colleagues and boss. Do that, and it is more likely than not that after some time your colleagues will find other things to gossip about. More than anything, do not go down to their level and do not engage them in any arguments. Good luck!