Two suggestions:
A)
React like your work begin / end is absolutely normal and obviously self evident (as it is).
Don't engage him in a way that lets him perceive you feel guilty, are irritated or "set back" by his remarks.
Answer non chalantly "yup" or "yeah", if you're feeling bold, let him know
you "thought it's a good time to work" or "gotta go, need my beauty sleep"
even joke about "you were on your way to the airport heading to (insert exotic holiday destination) but didn't want to leave him alone here"
... or any variations that convey that your behaviour is above reproach, start / end times are fine as they are, you're the "good guy" doing nothing wrong and he's being silly or petty.
Make sure you work your contractual weekly hours (bill accurately on your worksheets, gather proof in case of disputes).
It's even better if you do overtime every now and then.
Any discussion his remarks might trigger with him, your colleagues or your boss would basically end once you point the above fact out.
(obviously along with the flexi time scheme your company offers)
Also, the guy actually embarasses himself in front of people ignoring that conventional office hours are not standard / required in your company.
If you were doing overtime the previous day you may also mention something along the lines of
"and yesterday I did X hours" or "and yesterday I started at X hours and left at Y"
You could also point out
"it'll be a long day", "I'll stay later today", "guess I'll be here at diner time today" or something alike.
This emphasizes that you do your due time but it does validate his accusation somewhat.
Keep your reaction calm and reserved, without hesitation or uneasiness.
Once he realizes you're not phazed by his remarks he'll probably get bored and stop.
The above reaction is also really effective if your boss or other colleagues are present.
It removes any doubts about your working hours they may have after hearing his comment.
If you'd be appologetic, say "sorry" or react in a manner that may seem you know what you did is somehow not correct
he'd win by raising others concerns' about your behaviour.
B)
Another approach would be to have a private, frank talk, asking him why he thinks your times are somehow inappropriate.
Let him know (after his answer and if applicable) that you follow company policy, adhere to your contractual hours and do your work mostly at home, as it is allowed and that you don't appreciate his remarks.
I prefer A) but either way, if he doesn't stop and you're really getting annoyed or think, people may get upset with you about this you should go the formal route and include your manager or HR in your concerns.