To answer your specific question what you wrote is fine, but you know that. Were you asked something similar by a random stranger your answer would be perfectly acceptable. However, in my opinion there is a very different relationship between an employee and employer and answering this question in such a way can ruin or end the relationship.
It comes down to how much you value this job and how much they value you. If you really valued your position you would be creative and find a solution. You can make alternative arrangements for gym time to accommodate their requests. That might include joining a more expensive, exclusive gym that is not as crowded, or switching to working out in the early mornings.
If you answer in such a way as you propose, they might just say okay, we will find someone else. Perhaps you are so valuable to their operations that they will accept whatever you propose. However, very few people are so valued by their employer.
Given that you are a contractor it would suggest you are viewed as temporary and expendable. Sure, I have held some long term contracts, and have even survived layoffs of full time employees, but I complied with requests for "operating hours" and other demands of my employer. Not being accommodating to "normal operating hours" is not a place to be if you are looking for long term employment.
Perhaps a better course of action is to come up with options. Sure, I can attend those calls but I would like to take an hour of lunch at 10am (your time) to get my workout in when the gym is not crowded. Or I can change my normal working hours will be x to y (and you can work on your pet projects in the morning). Do either of those work for you guys? Is something else acceptable?
If you value this job, I would find a solution and be accommodating.
P.S. As an hourly contractor, I would not recommend "doing favors" an attending those meetings for free. Instead, I would flex my time. Hours spent on those calls should be reduced from your daily hours or paid extra. It is the nature of those types of positions.