It is somewhat disappointing that some of the answers thus far are assuming you are the issue or your behavior was indeed flirtatious.
I come from a different culture in the United States where gender roles between men and women are not as rigid and where 1:1 interactions between female and male colleagues are considered normal , with there rarely being any suspicion of improper motives from either party. I have went to lunch platonically both in groups and alone with these female colleagues , and never had improper motive accusations made against me by them.
I am assuming your chats to these 2 men are about innocent topics, such as how one's weekend was, hobbies, food / cooking and other topics one might talk about between friends. If so, and your tone was normal as you would use between platonic friends then the reactions of these two men seem to be an over reaction. If I was one of these men and had a family, getting defensive and assuming bad intent on part of the OP (that I wanted to be unfaithful despite having had a son, even though there is no evidence from OP that she intended to flirt with me) would not be my first reaction.
This answer is meant to recognize gender expectations are rarely useful, with instead, mutual respect / recognition of gender equality being the goal. Platonic friendship is not incompatible with workplace professionalism, and is not per se suspicious, not should it be interpreted as being unfaithful to spouse / family.
To conclude, it does not appear the issue lies with you (benefit of the doubt). Hence, I suggest you directly say to these men that if they construed your earlier chats as flirting , it was not your intention at all. Do not apologize as doing so can mean you acknowledge what you said was indeed flirtatious. You have done nothing wrong. If your behavior is not objectively offensive, such as by making unwelcome sexual advances, others misinterpreting innocent behavior between friends is not something you can do much about.
If these men continue to question you after you have made your intentions of friendship clear, I strongly recommend escalation to your manager. You have the right to not have others interfere with your work, especially in non work related matters. It is in your managers interest that you feel comfortable interacting with your colleagues whom you need to, and your manager , assuming he / she is good, should be supportive.
I know India may have societal expectations on gender roles, but remember you have rights as an individual in the workplace. Do not be afraid to be forthright and assert yourself to others who are interfering with you doing your work if needed be.
Response to comments
I acknowledge that there are instances in which significant alone time with colleagues of the other gender can be for improper motives, but I don't see anything in the OP question that she intended to be flirtatious or were otherwise intentionally ignoring how her behavior could be interpreted.
Yes, I am aware of the MeToo meme and cases such as Harvey Weinstein , but these cases are the exception to the norm. Be cautious , yes, but don't restrict your social interactions just because there is the possibility of an untoward event.
This response is not be condescending but to show if one acts normally , social interaction between men and women in the workplace is not by itself suspicious or unprofessional. It's not to call out the Indian culture as backwards either.