forget for a moment that it was a party. Lets say the employer wanted some people to have current CPR training. To encourage people to get trained they allow employees to spend 2 hours in a class, while everybody else has to work. is this legal? Yes it is.
The employer by allowing the party goers to leave work early is doing something similar. Yes the party is more enjoyable fro some people, while others would perfer the CPR class.
If the party was for everybody, then those that chose not to attend should not expect to be given 2 hours of leave.