There's always going to be a tradeoff between "everyone can manage their work/life balance however they want" and "we all need to be able to communicate to work efficiently". With the capabilities of the modern workplace that doesn't always mean that everyone has to work in the same building, at the same time 100% of the time. But those are the tradeoffs, and thing to do is to figure out the balance that works for your company and what you need to get accomplished.
The reason to drag people in and ask for a set of core hours is generally that they are hard to find, and difficult to get information from. I'm assuming, since you said productivity is good, that your hard to find people are not having issues getting things done themselves. If that's not true - if you don't see evidence of their work on a regular basis - then this is a different conversation about getting work done and being accountable to deadlines.
I like the idea of core hours, but my thought is that it's best to discuss it as a team, particularly since you are a small and flat organization. Ask for times when people can promise to be available for meetings and discussions, and verify that this is sufficient time for the group. 11-2 with implications of a lunch break is a nice bare minimum in the US, but mileage will vary. For example, I noticed in some places, people go home for lunch, so lunch is longer. In other cases, avoiding rush hour traffic may be a particular driver.
At least if you figure it out together, you can make a group decision and get opinions.
One decent heuristic for this style of work is "if you are late to a meeting you accepted, you are late" - in other words, be a grown up and manage your schedule. It's OK to miss core hours for doctor's appointments, illness or other issues, but unless you are sick, don't skip a meeting - plan accordingly. Don't accept a meeting when you have a doctor's appointment or need to be home for a repairman. But don't decline meetings so often that the team can't get a hold of you.
The other trick to increasing productivity is knowing what you want to accomplish. Sounds simple, but it's not - is it quality? faster deliverables? more stuff? Know what you want, and that WILL help you figure out what work patterns are needed to get you there.