If they've said (ideally in writing) that they will not return the documents unless you pay them, that's extortion and should be reported to the police.

If they've not actually said this, they've not yet done anything criminal and if they were approached by the police could claim they had intended to return your documents, but hadn't got round to it.

You could reply to the mail with something like :

> I acknowledge receipt of your mail of [date]. Please enclose a copy of
> my agreement to cover your costs when you return my documents.

You're acknowledging that you've seen the mail - not that you're agreeing with anything in it. If they're intending to keep your documents, your response is likely to trigger a reply that you could take to the police as definitive evidence.

If they're as chaotic as they seem, they might send you something to sign saying you accept their costs when they return your documents. Don't sign it, and tell them you don't accept liability for their costs. By this time you should already have your documents.

If they are able to produce something you've signed to say you'll accept costs (read the small print in your agreement with them), you may be stuck with paying. But at this point you should have your documents.