Cost. Employee compensation is far more than office rental costs
The problem is that your employee headcount can be scaled up and down easily, but your office space can not.
Our company employee count is steadily growing over the past few years. Unfortunately our buildings stay the same. So it gets more and more crowded. Offices planned for two people get staffed with three or four, conference rooms get converted to offices and walls get torn down to fit in more desks.
There are of course alternative solutions. Add an additional wing to an existing building (if space allows). Move to a larger building. Or acquire an additional building and move some departments there. Unfortunately these are feats which require a lot of planning and will be disruptive to the company. Also the cost of moving to a different building should not be underestimated. So stop-gap measures like converting floors to open plans can look attractive to the management.
Misbeliefs that open-space offices are more efficient.
Unfortunately that misbelieve is still very ingrained in many managers. That's because it isn't such a black-and-white thing. There are some people who can work well in open plan offices. There are some tasks where open plan environments are more efficient. In many cases, these are the minority. But these counter examples can make it difficult to make the decision to abandon cubicles in favor of isolated offices.
Also, managers do not want to run behind every single new trend. Open plan offices used to be the way to design office buildings. Now it's suddenly private offices. But maybe that's just a fad? Do you really want to spend good money on what could be obsolete again in a few years?
Common wisdom needs time to spread.
while companies that make smart decisions are more likely to succeed
They certainly do succeed in one area: employee acquisition. When I am looking for a job, I always want to know about the work conditions. If they can not promise me that I won't be working in a room with more than three other people, then I won't take the job. I am not alone in this regard. The Joel Test includes this criterium too. "8. Do programmers have quiet working conditions?".
But hiring the right people is not the only thing which decides about success and downfall of a company.