If you get a phone call from an interviewer about a position you have applied for, and you miss the initial call which regards them telling you that you have been shortlisted and are eligible for an interview, and then subsequently leave you a voice message to tell you; and you subsequently call them back later in the day, is that grounds for a black mark to your name?
It should not be a "black mark" to your name if the call was unscheduled. Most people will expect you to have a life and not be waiting anxiously by your phone.
In a way, this is an opportunity to carefully listen to the message, review the materials you've been provided and refine your research before reaching back out to your caller. In a small way, it can be used to give yourself a slight advantage over a shortlisted candidate who unreadily answered the phone call.
I'm going to answer both sides of this question - for both recruiters and candidates:
Recruiters and interviewers work the same office hours as everyone, and should expect to get a voicemail instead of a real person when they make an unscheduled call (given that you possibly sent your application in days or even weeks before they call candidates, there's no way you could know when they would be making the initial phone contact). They also shouldn't expect to be able to ask more than a couple of questions in this first contact ("are you still interested/available?" and "would you be available for a phone screening or face-to-face on such dates?").
Calling them back either later in the day, or early the next business day, is a reasonable response time. For similar reasons to you not being able to answer at any given moment, you are likely not able to respond at any given moment.
I'd not leave it longer than the next business day though - the interviewing process is an annoyingly fitful one from an applicant's viewpoint (open ads for weeks, then contact all potential candidates in one or two days, then wait for all of the interviews to be conducted, then contact the successful ones for further interviews, then wait for all those to be conducted, then "quick please make a decision on our offer").
I'd also make sure that my voicemail sounds entirely professional (if you can override the network's default), and don't promise any response faster than "as soon as I am able".
Even if the recruiter/interviewer emails you following the initial phone call, I would still call them back as a courtesy (there's a good chance the email was just a blast from their job portal - and you calling back will stick in their head), as well as respond to the email.
The general rule I was always taught was to return a call/email from the morning in the afternoon, and a call or email in the afternoon the next morning.
Now, of course, if you get a call at 8:00 am, returning it by 11 or 12 is not going to make you seem desperate, the same thing goes for getting a call at 12:30 and returning it by 5. As a general rule though, it's a good rule of thumb
It is not seen as unprofessional to do so because everyone is busy. Answering too quickly may make you seem desperate or unprofessional. I'd be asking myself why you had so much free time, for example.
As an aside: In general, don't be too concerned about losing "points" when job hunting. If they want you, you'll have to do something a bit out of the ordinary for them to pass you up. Job hunting is a stressful time, don't add to your own stress.