I have a friend who is now 57. He joined a company when he graduated from university and never worked anywhere else. Now the company is shutting down. He's been there over 30 years. Some people are holding that against him, a little, because he doesn't know how things are done in other companies, doesn't have a wide range of experiences. So I'm not going to say that nobody will ever think less of you for staying with one company for a long time.
But 3-5 years is not that length of time.
What's really more important than optics is reality. If staying with this company is easy and comfortable and doesn't challenge you, that's fine. You just need to understand that easy, comfortable, and nonchallenging is a choice. Leaving to go to a different company where you have to learn new things, prove yourself all over again, and start fresh with new people is another option. It may be very rewarding: you may rise faster throughout your career by being the person who's never satisfied, always learning, always pushing yourself to do more and be more. Or, it may be a treadmill of competition and offbalance "adjustment periods" where you're always telling yourself that things will settle down soon but they never do. It could go either way. And over a 10 or 20 or 30 year period, you can choose to be, or realize you are being:
- the one who stays put and stays comfortable
- the one who always reaches for new heights, then for still higher ones
- the one who is never at home, never really leading or contributing, always moving on
When I look at hiring someone with 20 years of experience I want to know which of those you are now (it can change over time of course.) So more than worrying about what people will think of you for not changing jobs, or for changing jobs, let me ask you: which of those three people do you want to be? Once you know that, you can start being that person and ensuring that your optics also reflect that you are that person.