I personally think that skipping levels in such a small team won't do any harm.
First thing, a 20 person is not small.
Secondly, it can totally do harm.
In most environments, the team leads are responsible for being aware of what the team is doing, keeping it coordinated and removing any impediments. One of the most common impediments to software teams is project managers coming by and making the individual team member report (again) why they're not done with XYZ rather than actually working to get XYZ done.
If your teams do SCRUM then attend the scrum and hear what the impediments are. You can ask short questions or have a quick post-scrum to understand better what the issues are. If your team uses email, you can certainly ask to be CC'd on discussions of problems. And you can absolutely volunteer to help remove impediments.
But if all the engineers see you doing is being yet another person asking them about their status (read: not helping to solve a problem) you become a problem to avoid rather than a resource to help.
Another problem I've seen is when project managers try to give instruction. "Hey, why don't you work on bug F?". The team lead has probably already given the engineer instructions about what to work on. Changing that is setting the engineer (and by extension, yourself) up for failure. Now the engineer needs to decide which of the two orders to follow. Some will try and do both, leading to failure. Some will just shut down and not do either, leading to failure. Some will follow the team lead, harming the relationship with you. Some will follow you, harming the relationship with the team lead, and possibly blowing up a well coordinated plan. And some will just call a meeting so that you and the lead can fight to the death (or otherwise resolve your differences).
So no, I don't think it's healthy in most environments for project managers to by-pass team leads, but it's not a matter of hierarchy. It's more a matter of coordination. If you leave the team lead out of the loop, they can't do their job and the individual engineers can't focus on doing their job efficiently.