As you said, it is considered a red flag just like anything else that potentially puts you in a negative light. The word potentially is critical here. I means something to investigate further in many cases or possibly most case it means someone to investigate deeper or someone not to bother with if there are other well-qualified people who do not a potential negative who have applied.
Getting hired is a competition, your qualifications are compared to those of the other applicants as are you personal traits. Having a gap may not throw you out entirely, but it almost certainly lowers your ranking in the minds of those evaluating potential hires. If it lowers it enough, you won't be asked to interview and you lose the chance to sell yourself.
AS to why gaps cause people to lower the ranking, there are many reasons.
First it means you have less experience than others of roughly the same age group. This often a problem because the guy 5 years out of school with two years worth of gap time, wants to make the roughly same amount as his more experience age peers. It also often mean they apply for jobs they are not actually ready to do based on their current level of accomplishments. That makes the person a riskier hire.
Sometimes a gap means the person has health issues that the company may not want to take on. Or he may have spent time in jail. Or some other disastrous personal problem that may or may not have been resolved. That makes the person a riskier hire.
Sometimes it means that he or she is not well qualified even if the resume appears to be so. A lot of people with large gaps do not interview well and that is why they are not getting a job. This makes hiring officials then see a gap and think, this is person no other company wanted, so why should I? I have seen many interviews where the people who look well qualified can't answer easy questions about their own profession, so this becomes a concern when you interview a lot of people. That makes the person a riskier hire.
In some fields a large gap may men the person has lost touch with the technical knowledge of the profession. That makes the person a riskier hire. T(his is most likely to apply only for a current gap not a past one.)
More troubling to many employers are the people who choose to step out of the workforce for personal, but not family emergency type reasons. If you quit your last job to go hiking in the Andes, then the next employer is going to wonder if he is going to be left in the lurch, the next time the whim to go play hits. I have seen people who are able to do this and find jobs immediately when they are ready to come back to work, but only because their performance is so superior that companies are willing to take the chance. If you are not in the top 1%, then this is something that is likely to cause companies to lose interest. Especially if you do it more than once. It indicates not only a lack of commitment to a particular employer but a lack of commitment to working at all and that is just unacceptable to many hiring officials. That makes the person a riskier hire.
Because so many of the reasons for a gap make the person a riskier hire, hiring officials tend to take gaps seriously. How seriously of course depends on the individual hiring official, how hard the job is to fill (gaps are much less likely to be knockout factors when the job is hard to fill), how impressive the rest of the package looks and the relative ranking to other people who applied for the job. This is not to say people with gaps can't get jobs, just that it puts them at a disadvantage in the hiring process, so avoidable gaps should be avoided.