Guidelines
This is mostly a set of guidelines. There's no one right answer. Sounds like this has become an informal trend, and it's something to be aware of. Personally, I've never seen that either bucking the system (staying) or following the norm had a huge impact on my work, as long as I stuck to these ideas.
Don't advertise.
Is a common term you'll hear on this stuff. In general, management doesn't want to change the rule, probably because they want it to be an option that if things are getting done, they can ask people to stick to the regular hours. Generally "don't advertise" does not mean lying. If someone asks point blank, answer truthfully. But don't start the day saying "wohoo! I'm so glad to be leaving early!!"
It also means "don't count on it" - if you arranged trip tickets assuming an early exodus and then have to stay late, this isn't an official policy, it's a courtesy - so don't presume it'll always be true.
Contract vs. Permanent
If you are a contractor, this become hairier. I'd advise talking explicity with your boss if this is the case - lying about your hours is not OK, so either work the full day, or claim fewer hours.
If you are a regular employee, you have a bit more flexibility, and may be able to run with the herd on this.
Make sure the work gets done
This is the bottom line for just about anything. Don't leave early if your work is not hitting it's deadlines. Even if you have a whole-team problem, put in the effort. Whether or not other people realize it, not hitting a deadline is a problem and sooner or later, the fact that folks are leaving early will come up.
The work getting done may or may not also include coverage of a time slot. It sounds like no one in your office is manning something that has to stay running - but in these situations, someone is on the hook to keep it running, and you don't want to be the guy that bailed when it was your turn to watch over it.
How flexible are the hours?
It sounds like some people come in early, and if the hours are flexible, others may be staying late other days or working through lunch to get the full minimum hours in. This is pretty variable, but it's an option in some offices. The trick is, you don't want to be the guy short on hours, if everyone is really putting in 40 hours, but in a flexible way...
When do the bosses leave?
In cases where I've seen an approved but unofficial exodus, the bosses make it a point to be the first to leave. It sets the tone. In formal offices, I've seen the descending echelon - Big Boss leaves at 2:00, his subordinate leaves at 2:30, mid level managers at 3:00 and so forth... But even so, pay attention to when your boss leaves. If the whole office is dark, and it's just you and him, ask him if he needs anything and then head out if it's a "no", but otherwise, don't be on the early side.
Alienation
I often find that people feel more alienated than they actually are. If you're a good team player most of the time, then I'm going to bet that your tendancy to leave late is not something anyone is going to mind.
But - be aware if folks are making plans to grab a drink or play a game after leaving early - you are more likely to be on the outs from not participating in a social function than you are from staying to get work done.