I would try to get there, if possible, as it is big disadvantage not be on first meeting. Also it is good to know all those people beforehand and there may be some interesting / important information about your upcoming work, which later "would everyone just know" and so will not repeat it for you, until you explicitely asked (for what you do not know you do not know - so probably will found it only when you will do/not do something, which all other would be strange as "it was already discussed in so much details" (on the first before work meeting)
But if it is not possible (and it sounds it is on workday too), I would send letter/email/what your preferred communications is, that you are verry sorry, you cannot go there, because you are still bounded by previous contract and cannot affort (may or may not add both "finantially and in terms of time") to work for the new contract before its starting time (the date 2 month from now). (Well, as you are not expected to work for someone else, when your contract would start).
Anyway you should ask about what important was communicated there (jus after this meeting ends, or when you start actually work there - depends on so much details that we do not know, what is more appropriate)
I personally sometimes refused attend some team-building events as it was not on work hours, was not paid overtime and I had other plans for the time (and sometimes I attended it). I announced it (politely but in firm terms) as soon, as I got the invitation and all went ok and nobody had problem with it, as I announced it as soon as was possible, so it was not unexpected at all. The event was not part of my work description after all. On other ocassion I happilly enjoy others on some off-work meeting/celebration/team-building, when I wanted to do so - so nobody was looking on me as "strange and always out of collective" kind of person - which I really am not.
When I rejected the offer, I did not specify, what kind of plans I had for the time, as it was my personal thing, nothing to do with company. (Sometimes that was important personal event, which I planned for and pay for couple month before, sometimes my plans was just to sleep and play games). All questions (and there was only few of them and not too serious) I answered with simple, but polite version "it was my personal think and it went well, thanks for interest" (and that was all about it)