This happens all the time. Constantly. It's "your loss" unfortunately and that's that.
Pay, politely explain that the work was disappointing, and move on, never use them again (unless in the future you change your mind).
Unfortunately all you describe is "the job was done poorly". All design work is subjective, and, (as with musicians!) it is completely commonplace and normal that designers will run over time.
If you're paying for (say) programming or house building to be done, you have specific rational goals, it's common with programming to "only pay for points A, B and C but not D" from a contract or whatever, since programming is rational and can be explicit like that.
Design simply isn't like that, unfortunately.
Just like with musicians, you pay 'em and if you don't like the result, it's a loss - try again with another.
Unfortunately that's the situation. I would strongly suggest absolutely don't compromise your reputation on the freelance scene by not paying. Unfortunately design costs are a "suck it up" cost.
Occasionally you'll pay, and just not be happy - it's a cost.
Design is usually the most important part of a project in reality; Designers have a tough life; they always get paid even if unfortunately the job went a bit soft and shitty: because it's totally subjective. And it's just unrealistic to punish designers (same with drummers!) for smoking pot, being a bit late, missing a meeting or the like. I'm afraid!
If full payment is truly not realistic:
What language to use with the freelancer? I suggest
"Since {extremely short description of the problem}, you can't realistically expect us to pay for this. Unfortunately we won't be able to use you again. As a good faith measure we will pay you a small amount of the fee on this project, no questions asked. What about 25%? Let us know your thoughts for a quick resolution so we can both go about our ways."
Regarding {extremely short description of the problem}, I'd say
"Since the work is unusable and it was four weeks overtime, you can't..."
The key IMO, do not get in to a discussion of the whys. Leave it as a fait accompli one-word summary. (eg, "unusable").