I would like to ask for it as soon as possible, but don't know how to
approach my new boss in a professional and non-awkward manner.
You are seriously overthinking this one.
Something like "Hey boss. I noticed that the women's bathroom doesn't have a sanitary bin. Could we get one?" would be fine. If you would find that too embarrassing to suggest in person, make the suggestion via email.
You might even offer to pick one up and get reimbursed. And as @Martijn suggests, you could even find one and order it online. In small offices it wouldn't be unusual at all for folks to order things then get reimbursed.
It's understandable, but try not to be nervous about this sort of thing. We've all gone through a "first job" experience. It gets much easier with time and practice.