In the UK, just as you can ask to take holidays, the company can ask you as well to take leave. That must be done with notice that is twice as long as the leave. So in principle, they can ask you to take one week leave two weeks from now. (Paid leave obviously). Where I have seen that is when a company closes down over Christmas, so they say "everyone has to take holiday from December 27th to Jan 2nd", for example, and they tell you many months ahead, and it's a time when many people want a holiday anyway, so nobody complains.
If you had 10 days of leave (two weeks), then they can ask you to take the leave 4 weeks ahead, before taking it. If they want you to take the leave within your notice period, then obviously you have to take the holiday two weeks before the end of the notice period, so they have to ask you six weeks before the end of the notice period. In that case they are in their rights.
If they told you 20 days before the end of your notice period, that's too late. The days are working days, just like the leave. If they realised they want you to take your leave 3 weeks before the end of the employment, then legally they could have asked you to take one week off at the end of your employment. (One week leave, telling you two weeks ahead). "Double the time of the leave" is counted in working days. And it counts to the start of the leave. So if the company wants you to take leave next Thursday and Friday, they have to tell you this Thursday (so Friday, Monday, Tuesday, and Wednesday are the required four days of notice).
So if they didn't give you enough time before telling you to take leave, then you can tell them that they didn't give you enough notice, and you are not going to take your leave. At the end of your employment, they need to calculate how much leave you are still owed, and pay your salary accordingly. Normally 1/230th of your annual salary for every day of leave that you are owed.
Re-reading the question: It seems your notice period is over, and it seems that no matter what they said, you didn't take that leave. You came to the office and worked. They could have told you "sorry, but you are on leave, go back home", but they didn't. So you didn't take leave, so they have to pay for it. They can make a stink about you not taking leave when you were told to, but that doesn't change the fact that you didn't take leave. And if they told you to take two weeks leave, two weeks before the leave, that's not enough notice, you didn't have to take the leave.