There are legitimate issues around trust and communications.
In general people in close proximity to each other will have more open lines of communication than those who require some communication tool (phone, IM, wiki, etc). Being face to face provides much better feedback for question tone as well as whether the answerer is understanding the intent behind the question. Also it is much harder to ignore a person knocking on your office door than an email or an IM. Speed and quality of communication are benefited from nobody being remote.
Trust is important as well. Being able to see butts in chairs provides some managers the perception that their workforce is in fact working. If you were inclined to slack off, it would be much easier to do so if you were not in the office.
With respect to your edit
For example, employer may start suspect people are under performing so work from home becomes the first suspect to blame for. Is that normal?
If I were in a managerial role on a team that was mixed, in house and remote and my team was underperforming I'd have to look at why expectations were not being met. If the root cause was poor quality communications (misunderstood requirements, people waiting on answers, work being duplicated due to lack of communication, etc) I would see getting everyone in the same space as low hanging fruit for improvement.