I see a very real downside - they know exactly what you look like and can now badger you at work.
I had a situation where an interviewee was doing very poorly, and when I interrupted him he banged both fist on the table and yelled at me not to interrupt him while he was talking and to let him finish answering the question. I stood up and ask "do you have any questions" as I walked to the door to get the HR person.
While nothing happened beyond a "can you believe that" story, I made sure I wasn't the last one out of the office that day. There's a 99% the guy just thinks he'll make a better impression on you, but a phone interview would have weeded out this guy, and avoided a really awkward situation.