People coming at a company are in search for a place to work, not in search for colleagues. Most often you can't have both of worlds. I mean you need to earn your living, and in a small company you could have the luck to find good people working on similar positions as you do, or instead some bad luck to find fellows that aren't of best quality as people. They are there also to make money not to become friends of you. Either way, the first matter that is considered is to be professional at your work. You are there to provide a good service to the company.
Now your question could be put as "do you want to lose your job or your 'friends' ?" Remember, they are your 'work friends' probably because you make their job somehow easier (so you both provide to the other something valuable during work time). I think this hasn't to do with the fact that the meeting is asked to be made in 'private time'.
My advice to you is to be sure that your 'friends' understand that this review might be similar to a work duty (at least in the eye of the employer) so that they must understand that you could express some opinions that are for the best of the company (not of them).
Even if I sound a little bit too harsh, making some 'alliance' with your 'friends' could be similar to stealing from the company (by accepting some bad behaviours from your friends part).
First of all, I strongly think that if your 'friends' are not good professionals you MIGHT NOT BECOME FRIEND WITH THEM in the first place, no matter what. This way you have nothing to regret about what you tell to your manager about their performance.
Second, imagine that starting from tomorrow you become their boss (so you create a company and hire them to work for you). Would you be the same relaxed person around them ? Would you accept to them any 'friendly attitude' as a kind of excuse for their lack of performance ?? I doubt about that! Or similar, you are put in a position of becoming THEIR team leader, and are penalized for any bad thing they do during their work. Would be you same good 'friend' as before?
In conclusion, they better first should deserve to be real friends of you, before you make worries about what you will say about them. And it's not important if they have other roles in the company - compared to you. Nor it's important that instead of having directly a review meeting with the manager he appointed some other person to be present in his place. You should worry nothing. Probably you should worry that the colleagues did not understand how well you do your job and maybe they give some feedback about you that isn't that positive (in your opinion). But anyway be positive, and try to receive their feedback with an open mind, in order to help you to become better if possible.
I would end with the remark that your own fear of others' opinion is the real sign that truly the most important factor at job is how good you are, and nothing else.
If I would hire people to work in a company I would ask them not to become friends (and stay in complacence together) but instead to educate each other and also learn from each others so that eventually all of them become as good as the best one from the team. And that best one will finally receive some good help from his fellows from now on, of course.
I remember of having in the office a fellow that had a bad character (the type of those stabbing people in the back, very eager to give bad opinions on the 'quality of code delivered' especially of people more qualified than him - behind their back, of course) so I suggested him that if he thinks he's so good, then go ask from the manager to put him as the project manager (or product owner, as it's in the software industry). He understood since then that he is very weak and not capable of assuming such role (or tell to better developers what they should do and how to do it) and since then stepped back from his toxic attitude. Most often people can't resist to a confrontation to their true self (not the others). Because the worst enemies are not the fellows around but themselves actually.