I'm a little bit unclear on a couple of points of the question, so please forgive me if I'm off-target. But, taking the questions (as I read them) in order:
1. Yes, it's absolutely OK to expect an on-site employee to let in workers that need access to the site.
The part of your story that is unacceptable is that the person (or people) in charge of hiring and scheduling this sort of work absolutely should inform the on-site employees who will give those workers access to the building.
If your office manager tells you that painters from Paint Contractors, LLc will be coming on Saturday around 11:00 AM, then you have enough information to know that the painters are expected and are allowed to be in the building by someone who has authority to grant them access.
It is not acceptable to have any on-site employee allow access to any random people that claim to have a reason to be there. If you don't have clearance from someone else to let them in, and you choose to do so, then in your employer's eyes you would be responsible for anything that happens, such as theft, etc.
I've been in similar situations, and if I couldn't verify with an appropriate person that these contractors were expected, I didn't let them in.
2. You may or may not be justified in feeling "unsafe" with strangers in the office. But that's not a decisive factor in whether or not they can be there.
There is little reason to think that a professional worker representing a company which has been hired by your employer to carry out a valid task in your office is any kind of threat to you. This, however, requires that on-site employees be informed of plans for such workers to arrive.
Indeed, expecting employees to grant access to anyone that claims they have a reason to get inside is the opposite of security-- it's not much different from simply leaving all the doors unlocked. As above, the lack of communication from your superiors about who will need access during non-standard office hours is a problem. The correct solution is to not grant access to anyone without prior clearance from management, full stop. It's trivial to convey that sort of information, and doing so improves security for everyone and everything in your office.
3. It depends on what you mean by "considered fine". It may be standard practice for your employer, but I personally would never operate that way.
The situation you describe is a massive security problem, even if nothing untoward ever happens. Leaving this sort of decision to employees' discretion, without giving them the information needed to make the decision, doesn't serve any good purpose.
One instance like this might be an oversight, but immediately afterward I would inform my superiors at the company that as a general rule I would not give access to any contractors without explicit confirmation from management that they were expected, the company they represent, and what they are there to do.