Yes, go.
but I'm not too happy that HR is forcing me to go sit in a room with the person.
That's exactly the right approach.
Also, realize a key thing that you did, which did not work out well for you. You wrote this in writing.
Bohemiam's answer has some positive aspects, but I'd remove the aspects of trying to deny. (Don't be dishonest.)
which was the recommended course by my supervisor
Your supervisor provided advice that may have sounded good, but it just did not work out good for you. (Whether this worked out the way your supervisor intended, or not, is unclear. Maybe your supervisor is more concerned about you being corrected, or maybe the supervisor was trying to do everything to please people so this would calm down. Unfortunately, it ended up being fuel, fanning the flames of a fire which has grown, and might grow further.)
Whether you're trying to communicate to the person in writing (on paper), in E-Mail, indirectly through HR, lawyers, etc., be careful not to communicate bad things in a way that will make an official record. This is why people say that when you get in a car accident or get accused of a crime, don't admit guilt to a police officer. (Even if you're guilty, especially don't do this on the same day.) [See footnote 1] Just remain silent, if you have to. Sometimes there may be surprising facts that are in your favor, and things could go better for you than expected, but those facts just don't count as much as an official record that works against you. So don't contribute to the creation of such official records.
In the future, try to apologize verbally (when you are presumably not being recorded), because then if the person does try to use things against you, they can only produce their reproduction based on their recollection of what you said. That may still be condemning, but often not as condemning as a self-created admission of guilt.
As for this meeting with HR, go. When I worked at a company with an HR department, HR was in charge of hiring, firing, promotions and other position transfers. Being on their bad side provides no advantages, so try to be cooperative, at least minimally. For instance, show up to this meeting. Even if your entire computer network goes down, show up to the meeting. (Make sure you have a co-worker back up.) And if the entire computer network does go down that day, tell the HR person that you wish to re-schedule due to the emergency that will affect the business. But let the HR staff decide whether to make the right decision, or to force you to attend that meeting at that exact time. Do what HR says...
... except, don't do anything to dig yourself in a deeper hole. Do not write additional material that signifies your guilt. HR may get involved if there is a lawsuit or an apparent potential lawsuit. Depending on the scenario and details like where you live, you might get sued personally. You shouldn't have to do anything to cause yourself potential further trouble, so don't. You might be required to sign a statement where you acknowledge that statements or other facts have been presented to you. Fine. Avoid that if you can, but if you must, then do. Don't create new material that could be used as an official statement made by you.
If people ask why you're not cooperating with trying to appease the emotions of the other person (a.k.a., make them happy), you can explain that you do not wish to continue using this formal process any more than what is required, as increasing the formal records (related to this matter) is unlikely to be personally beneficial to yourself. Any genuine concerns about trying to "make up" are preferred to be done without a person who represents any role of authority over you.
At this point, I suggest you may want to have a witness if this heated topic will be discussed further, but have this biased person be someone who is friendly to you, and make sure there is agreement that people won't be recording what you say against you. If this person comes to your IT office, let the person know that you won't be discussing this with them alone. Neither alone, nor around someone who is expected to make a formal record of the conversation. Such scenarios are too risky of generating additional problems for you.
[footnote 1] - If you have the time, check out video (Youtube) - Regent University School of Law - Don't Talk to the Police [time-jumped to 26min19sec] showing a class where a professor just gave a bunch of reasons why not to talk to the police, and then a visiting police officer provides his opinions about how much he agrees with that stance.