I'd have expected that all employees are expected to do what's best for the company.
No offense intended, but that is extremely naive. It ignores the entire history of labour relations, well, everywhere. There are literally thousands of examples and millions of workers involved in widespread organized worker actions designed to punish the companies they worked for. Do a search for "Lordstown Syndrome" for many examples.
You should expect that people do the behaviours that are incentivized and avoid behaviours that are disincentivized.
people seem fixated on hitting their KPIs as opposed to doing what's best for the company
Since they are punished for doing the latter and rewarded for the former, it's not clear why you expected otherwise. Incentives matter.
I am reminded of a question on another site -- posted by a professor! -- about why so many college students are fixated on their grades, rather than their mastery of the material. Because they've been told their entire lives that all that matters to their future ability to live comfortably is their grades. Why would we expect them to behave any differently?
We built these Skinner boxes; we should not be surprised when the human animals in the boxes try to maximize their rewards and minimize their punishments.
I've seen situations like these happen
Oh good heavens I was at MSFT for 16 years. I could give you war stories all day. X reports to Y, who reports to Z. X is doing poorly because X lacks skill B needed by Y but has skill C, not needed by Y. Q -- who does not report up to Z -- needs someone with skill C and not B, so Y moves X to report to Q, opening up headcount for Y, who hires rock star with skill B. Everything is great, right? Everyone is doing the best for the company. Z tells Y, "what on earth are you doing? We need a goat (low performer) on our team so that we can shaft one of them at review time, as required by HR. Give the rock star a bad review; they're the least politically powerful."
Similarly in the second situation, Bob should not care that his KPIs will suffer, but rather that the company benefits if Bob passes the job on, since the client gets a better product.
Again no offense intended but be realistic. Would you do that if you were Bob? I wouldn't! The supposition that Bob will take one for the team and be happy about it is not realistic. Bob will quit and angrily badmouth his former company on social media.
How can a manager stop his or her employees / branches from competing against each other at the expense of the company?
Give huge bonuses for cooperation that leads to good outcomes, and fire people who sabotage their coworkers. Huge bonuses. Like absolutely enormous. You know, like the bonuses that CEOs routinely grant themselves at the expense of the company.
Can this problem be tackled at the HR level ("hire people who don't compete with each other")?
One day shortly before I left MSFT they gave all of us personality tests, because middle management was obsessed with faddish metrics. Those personality tests showed that, SURPRISE, the people who were hired to build code analysis tools in a highly competitive environment were (1) analytical, and (2) competitive.
Being competitive is great. People who enjoy competing should be hired for competitive industries! The trick is to incentivize competing with your competitors, not each other.
A question you did not ask but seems germane:
Should companies incentivize actions that improve the bottom line?
Yes, but to a point. A case that comes to mind was of a rock-star-quality friend who worked for Twitter who was given the task of improve user satisfaction amongst users who post many photographs to Twitter, and made good inroads on this problem. And then at review time, they were reviewed on the metric how much additional advertising dollars were spent at Twitter as a result of this work? Since that was not the task they were given, it seems unfair to review them based on this metric; said person is no longer at Twitter and is adding value elsewhere.
The real incentive system should be:
- Management comes up with a strategy
- Workers implement the strategy
- Management is incentivized to come up with successful strategies that improve the bottom line.
- Workers are incentivized to implement the received strategy.
In my example of my friend, they should have been reviewed on whether or not the user satisfaction needle moved. Management should be reviewed on whether or not that strategy led to the bottom line return on investment.