However, I'm unsure now whether I would fit in well with the manager and his group
Think about how you could increase your confidence that you would fit in with the manager. Then try to move forward on that. For example, perhaps another talk with the manager would ease your mind. Perhaps a talk with some of your potential peers who already work for this manager could help.
You already felt that this manager was being rude by asking for your references before making an offer. Hopefully, you are able to get your head around that occurrence.
And it seems the manager isn't quite sure about you yet. Offering a temp-to-perm job makes it easier for both sides to give up on the other. I personally would never accept a temp-to-perm job. I've always felt that if both side weren't 100% confident, then it wasn't the right place to be. But then I've never been in a desperate situation.
In my experience, fitting in well with your manager is the single most important predictor of success and happiness in a job. An often-repeated saying goes "You don't quit your job, you quit your manager."
If you can't get to a comfort point with the manager, that doesn't bode well for your job at this company. Make sure you keep your other prospects moving forward until you have completely decided on this one.