In the UK, can a company deny explaining the reason why a prospective candidate was rejected, just because it allegedly uses a proprietary system to vet job seekers?
The company doesn't owe you any explanation whatsoever why it is rejecting you. If you suspect they are rejecting you because of illegal discrimination, you could take them to court - but proving that would be almost impossible unless the company acts totally stupid.
In court they can't refuse to tell why you were not hired. Your risk is that the interviewer might say "Mr. Ajram had spinach between his teeth from lunch, which made me sick, so I didn't hire him". Which would be a perfectly legal reason not to hire you, and would mean that you lose the case probably with substantial cost.
Others noted that you can ask the company to provide all the information they stored about you. They have no obligation to store anything about you. They might even be allowed to respond "we received your request, we deleted all information about you except a note 'interviewer recommended not to hire', and this note is all the information we have stored about you".
Under the Data Protection Act, you can request access to e.g. interview notes (e.g. see https://www.theguardian.com/careers/careers-blog/what-data-employers-hold-on-you-philip-landau). For government related employers like BBC, GDS it should be easy to get these data. For private sector employers I'd guess your mileage will vary.
However, I don't think you can force the company to give a reason for the rejection (although I'm not a lawyer).