The people who are already hired are in charge of the interview. They determine the priorities of the people there, who may or may not leave according to whatever reasons they decide.
I've been involved in interviewing people, and often some of the interviewers cannot stay for the whole length of the interview. That is because of some other task that the interviewer must attend to, and has nothing to do with the person being interviewed.
As for how to respond to what seems like games: I once worked for a company because they bought out the previous company I was working for. The owner once commented that he liked the interview questions used by the company they bought out, which were much better than the admittedly-bad questions of their old written-interview process. As an example, one of their old interview questions was, "Star Wars or Star Trek?"
They were actually a very great company to work for, but just had a lousy interviewing process. And they knew it. But fixing it just wasn't a huge priority for them. Judging the company by their bad interviewing process would have only led to many misjudgments, because their lousy interviewing process was not an accurate reflection of what it was like to work for them. Still, they recall someone who responded to several of these "off the wall" questions by simply answering, "Ask relevant questions." As you can imagine, it was decided that person just demonstrated they weren't very interested in being a team player who would cooperate with what they were asked to do.
Imagine you were interviewing someone. They didn't like what you were doing. Do you want them to disapprovingly challenge what you're doing? Does it matter much to you why they don't like what you're doing? If you are the person being interviewed because you want something, then it is probably just best that you be very cooperative with their program, even if something you don't like happens.
A real scenario: I remember once being in an interview. The owner of the company and his lead manager spoke to me for a while. At one point, the owner of the company left to get something (maybe some papers). He remained gone for at least a few minutes, perhaps several. I figured I probably lost him, and the interview may have effectively been over. I wanted him around, but I cooperated with the interview scenario. I kept interviewing with the lead manager.
To this day, I still don't know just what the owner was away getting. He was already part of the company, and he made the determination to step out, for whatever reason he determined. As a person who had no authority over him, and did not know his schedule, and did not know what other emergency task might have priority over the meeting, it was not my place to challenge his decision to step out (no matter how much I ended up not liking that at the time).
Later, the owner and I ended up getting along very well. Within about a few weeks of when I got hired, the technical manager (who interviewed me) unexpectedly needed to quickly depart for medical reasons related to a family member, and I quickly rose to effectively become the number two person in this company. I never bothered to ask the owner about what he went to go get during that interview. It just simply didn't matter.
Much later, after I successfully got the position, I spoke with my other interviewer, who had been the lead manager (before he left the company). The result of my interview was that right after I left the interview, the lead manager told the owner, "In my opinion, we can just stop interviewing other candidates right now. This is the right person."
He also let me know what I said which led to this decision that I was the perfect person for the job. I remembered that point of the conversation, and I remember it was during the one-on-one portion, when I was alone with the tech manager. While I was busy winning over that person, I was silently wishing that the owner wasn't missing out on that excellent part of the interview which was making me sound very good. Still, I wasn't in charge.
Keep using your time in the interview to try your best to look like the wonderful match for the position. Maybe your time alone will be a chance for the remaining panelist to ask you about a topic that the remaining panelist cares most about, but is something the panelist didn't want to bring up with the larger group. The reason doesn't matter that much. What does matter much more is that if the interview is still active, it is still a potential opportunity. Addressing what the panelist wants to know could be your best chance to sway that person onto your side. When the interview is over, you want as many people saying as many positive things about you as possible.