As others say - a common metric is 1 phone screen, 1 in person. More than that if the role is fuzzy or the company is hiring outside of it's normal hiring wheel house or trying for something particularly complicated. For example, I see more than 1 on site interview on first hires in a while at small companies, manager positions (especially if normally managers are promoted from within, or this is a new level of management/new role), or new jobs entirely (first UX person, first tech writer, first dedicated QA, etc).
1 - Know your needs
You'll end up dragging in candidates more often if you don't know what you are looking for. You'll save both your time and the candidate's if you have a clearly spelled out list of qualifications. The job skills are usually pretty easy - the people/team skills or professional qualities are often harder. Have an image of both. Don't let it be "a good fit for the team" - you'll debate on that endlessly and bring the candidate in too many times.
2 - Know the market
This is as much about how many different people you need to talk to in order to feel confident. Get at least 3-5 different candidates to consider so you can make tradeoffs and consider best fits. It's much harder to see if you have a standout candidate if you only look at 1.
This can mean letting one wait for a long time. That's probably better (but risky) than dragging the poor candidate in every week.
3 - Have a good pool ready on the first visit
It's harder to get good feedback from a group of people who all see things the same way - get a diverse enough group to get useful input and then book a full agenda. By "diverse" - I don't mean the categories HR is thinking of, I mean you want people in different roles, different teams and/or with different view points. I like having a hard quiz-giving guy and a guy who really wants a whiteboard debate on a meaningful piece of work. I want the guy who is annoyed by almost everyone, and the guy who likes everyone.
4 - Focus your energy on the quality of the interview and getting the best feedback as quickly as possible.
Interviewing is a huge cost and a huge distraction. Lessen the pain and get answers to fit as quickly as possible. Get a sense of what each round of screening should rule out. For example, a common format is:
- HR/recruiter checks out simple stuff - employment eligibility, general interest for salary/work/title, basic background verification
- technical phone screener - does this person have the basic skills, is there any reason to disqualify/not waste our time?
- interview day - a diverse set of managers and peers, and hopefully adjgoining groups see if this person meets expectations.
If you hit that point and you don't know - ask yourself strongly - what will get you the information you need? Should it have been asked earlier? What can you do next time to get more efficient.