Australian Security & Defense interviews can be pretty intense, but questions like 'how many partners have you had' are generally not of any importance.
The things that they ask you about relate to
- Your financial situation (Because they need to be sure you are not easily bribed)
- Your family situation (Because they need to know that your family can not be easily manipulated into betraying information about you/your work)
- Your acquaintances (Because they need to make sure your associations are not going to cause an issue EG: you're friends with a king pin.)
Where do they get their info:
As for where and how they get their information. A lot of the info they access is purely financial, and may include exact information about your bank balance, loans, mortgages, car payments and anything else that involves financing.
This information is always accessible to the government for a plethora of reasons, and is really nothing to worry about, it's a necessary evil.
The information about the people you're associated with may come through several sources, but is generally based on information you provide (EG, they will give you a form and ask you for the name, address and phone number of all of your relations (Parents, siblings and dependents). Then this info is checked over to make sure they are also not closely linked with anyone 'uncooth'.
How much privacy are you giving up? None that you will really notice.
How intrusive is it? It's kinda inconvenient trying to get everyone's names, addresses and phone numbers onto a form, and they may also be called to check details, so it's inconvenient but not really that intrusive.
You're not likely to get into a really high security position without having been through this process before, and they get a little more lengthy each time but the reasoning is the same. So for a general (level 3) security clearance, it's as described above, but if it's 'Secret' clearance or similar then it's a bit more in depth than that described above.
And finally, they may ask you about any business dealings that you are involved in, and people you are involved in businesses with. All the questions really revolve around "How easily could this person be coerced, corrupted, or manipulated".