Oh boy, all because of a damn doctors' appointment.
No. Not just over a doctor's appointment. This seems to be a variety of issues about your work ethic, and you're currently on course to be fired for cause. That will happen unless they see readily visible change.
But yes, the very obvious work ethic problem is your habit of notifying the employer about the appointment only hours before the appointment. You should have notified them the moment you yourself were aware of the appointment. No one wants to keep you from needed health care, but the last-minute notification of each appointment makes it appear to be excuses for some other activity. If you had notified as early as possible on the other appointments, the late notice on one would not have been so concerning.
On the hospital visit, since you knew or reasonably should have known these could be lengthy, you should have told them the moment you knew, that this could be an all-day thing.
This feels like they're planting evidence, to use as a pretext.
No. What's happening is your indifference to the company's needs regarding your medical appointments is causing them to examine your work ethic, and they are now examining your other work in that light. You will need to suffer more of this scrutiny for quite awhile (or until fired), so sharpen your CV and/or work really hard on your engagement to the job.
I promised to take more notes to remember stuff we talked about.
Well that sounds faked, like things people say when they don't really believe what they're saying but feel like they need to say something. And that will have been noticed by your boss and manager, and reflects further on work ethic.
I don't think you and your boss are communicating very well, and that's something you want to correct ASAP if you want to stay.