I have a boss who is often not in the office, sometimes for weeks. But I and my colleagues still do our jobs.
We all know it's his company and he provides the work and he pays our salaries. As long as he provides enough work for us and pay our money we don't really have to see him. And if we have some questions we email him or call him and we get answers. So we can do our work without him being present.
Sometimes the boss is at home (and works?) and sometimes he is traveling to see customers. How much of his day he really sees customers is another question. But nobody really cares as long as he gets his work done. And his work is providing us with work and making sure we get paid.
I don't know if this will work for you. It obviously depends a lot on your work. If your employees could do the same without you and make double the money then probably you should not leave them alone - otherwise they will do it without you.
But if you do your job of making sure they have work and get paid and you are available when necessary then it shouldn't be a problem if you are not so much in the office.