However as the company grows (expecting to double in the next year) and beyond, how can I ensure that this policy is kept to/isn't abused when extended to a larger and larger amount of people?
This is what middle management and company culture are all about.
As new employees join, make sure they understand your liberal company culture - what kind of trust they are entitled to, and what is expected of them in return.
Then, as your company grows, make sure all middle managers understand this too. They must hold folks on their teams accountable. If abuse starts to occur, the middle managers must talk to the relevant employee, remind them of the rules, and help get them back on track.
At some point your company will grow beyond the point where your personal span of control can be effective. So your challenge is to find and train middle managers to carry out your company's culture effectively.
Flexible work hours can be very appealing to current and potential employees. You are right to be concerned that it might be abused - in my experience such abuse can ruin a great culture. But the way to avoid abuse is not by imposing formal control on an otherwise trusting culture.