Statistically, he's wrong.
Videogaming is associated with 11-year-olds, so let's look at this 11-year-old study of World of Warcraft players in its heyday. And this is by Nielsen, the big name in TV view ratings for the last 50 years, so they are a reliable source.
knock me over with a feather
You may scoff at 2008 World of Warcraft. But it was serious commitment with a high cost of entry: a competent PC (even new low-end PCs won't do - check the specs), $15/month, and back then, $50 buy-in. And you have to admit, the time commitment was pretty hardcore. Today, games are much more accessible, but even more addicting.
Almost everyone has a modern phone that will play most games. The "Free To Play" business model reduces cost-of-entry to nil, but is psychologically engineered to be much more addictive than WoW. So the time commitments get higher and higher. (Again your definition of "Hardcore" might not be time based, but you have to admit, that's fair).
All that to say: tossing out the bogon of "Only kids play video games" is so wrong that he ought to know better. So he is either an idiot, or he trolled you.
The real problem is, it's a hole where something should be
They asked that question to learn more about you as a person. And the problem is that "spend my time gaming" is a null answer, which says almost nothing about you... but it also means, there isn't much more to say. And that's the real problem.
They're looking for things like
- I'm a volunteer coder with the Firefox project
- I play football in league play (my favorite team is MY OWN)
- I'm on the Board of Directors of the local library
- I maintain a well-used Perl module in CPAN
- I helped turn the local WWII museum into a financial success
- I buy distressed houses, fix them up and rent them out
- In this MMO game, I am the guild-master of one of the top 3 guilds on my realm.
- I volunteer with kids at the local hospital
- I'm restoring a '71 Barracuda
- I've worked with neighbors and the city council to beautify our street
- I live on a canal boat and do continuous cruising
- My family is my life
This isn't just interesting lore to make smalltalk. This is really about serious questions like: Do your activities reveal a hidden capability from you that might reflect an ability to grow in the job? Are you stable in the community? Do you have an appropriate life-work balance?
You whiffed all of them. You left the impression of a person who has no life outside of work, creates nothing, has no passions, and just sits in a chair in front of a screen, consuming media. That doesn't make you a bad employee, but it makes it hard for them to see any potential in you.
You might give your job 100%, but it's 100% of not a whole lot.
By contrast: Joe runs a top 3 guild which was the first on-realm to down Xorragath and the 19th worldwide. You'd definitely call that person a hardcore gamer, yes? But Joe is so much more. Joe had to manage the guild - recruit the right people, expel the wrong people, keep the politics contained, keep people showing up, and manage progression. I.E. knowing exactly when the lower dungeons have given you enough gear to advance to a tougher one (without frustrating, time-consuming wipes), keeping people educated and on-focus for the unique challenges of each fighy, etc. That tells management that Joe is probably ready to lead a developer team, because Joe can see the core objective, and choose appropriate work flow and targets while keeping the team on track.
Joe isn't going to coredump all that in the interview. But when the interviewers here "guild leader" they'll hear "leader" and start asking what that is and means. And Joe will open up and talk about the things I mentioned, the project management and personnel challenges. And interviewers will think "Yeah, that sounds like managing a team around here. OK, good to know Joe could go that way. We'll see."