The below is all assuming this colleague who wants to be part of the meeting actually is part of the project that the meeting is about. If he is not, then simply say "sorry, this meeting is for project participants only", and leave it at that. If this colleague insists on attending the meeting despite not being part of the project, then have a chat with his manager: "Joe is not part of my project right now, but he asked to be a contributor. Can we have Joe on our team?" Then if the manager says "sure", you have an extra developer to assign tasks to, and you should expect the colleague to contribute equally to the other team members going forward (and if he doesn't, then you should report him to his manager for insubordination). In the vastly more likely case that the manager says "No, Joe has his own work to do", you forward that message to your colleague: "Sorry, I asked for you to be added to my team, but your manager declined. As a result, I'm not going to invite you to this meeting".
If he is part of the project, then he should be part of the meeting. If he is not part of the meeting, then he may be missing some context he needs to be productive on the project. Even worse, he may get second-hand information from the meeting from someone else, and then he will be misinformed, which is worse than uninformed.
I would just invite him to the meeting. You may want to forewarn him that the other guy who you think he will have a conflict with will also be there just so he's not blindsided. As the organizer of the meeting, it is your responsibility to defuse or deescalate any conflict that might come up between these two as a result of them both being in the meeting. It is also your responsibility to ensure that all relevant project participants have the information they need to work productively.
If there is a serious conflict during the meeting, you should raise it with the manager(s) of the people involved in the conflict after the meeting. It's not an issue of not liking someone, but at work you have to be professional, and if these two people are causing conflicts, that's not being professional and their managers should know about it.