I previously worked on a project (Alpha) at my present company. Due to personal reasons, I was on leave for a few months. When I returned, I was assigned to another short project (Beta), which is scheduled to end in about two months.
The previous project manager has informally asked me to rejoin Alpha after Beta ends. I asked for some time to decide. I do not want to return to Alpha because when I worked there, I got average ratings despite working hard/smart (at least in comparison to the others). I found the team to be a bit toxic as well. I do not want to tell them these reasons.
In my company, project allocation is handled by the HR. If they cannot assign me another project after Beta ends, I will be "on the bench" (on payroll without a project) for a couple of weeks. Hence, I do have the option of refusing the project manager's request.
How do I politely decline the invitation to return to the previous project?