If they've said (ideally in writing) that they will not return the documents unless you pay them, that's extortion and should be reported to the police.
If they've not actually said this, they've not yet done anything criminal and if they were approached by the police could claim they had intended to return your documents, but hadn't got round to it.
You could reply to the mail with something like :
I acknowledge receipt of your mail of [date]. If you have a copy of an agreement where I accepted I would cover your costs, please enclose it when you return my documents.
You're acknowledging that you've seen the mail - not that you're agreeing with anything in it. If they're intending to keep your documents, your response is likely to trigger a reply that you could take to the police as definitive evidence.
They might send you something to sign saying you accept their costs when they return your documents. This will show that they don't already have this. Don't sign it, and tell them you don't accept liability for their costs. By this time you should already have your documents.
If they are able to produce something you've already signed to say you'll accept costs (read the small print in your agreement with them), you may be stuck with paying, though it would be worth consulting a lawyer if this happens. But at this point you should have your documents.