You have a cultural and organizational problem there. People are used to having you at their beck and call, and proper communication channels are neither institutionalized nor set up.
If you are indeed their only Sysadmin, obviously with such a high volume of calls you won't be able to do any reasonable work toward improving things during work hours.
If you have indeed already have a help desk department, I am afraid you will have to enforce proper communication channels and escalation procedures. Stop answering the phone, or pass that phone number to the help desk team. You might also have to be ticker skinned and enlist an higher up help, as people might leave their desks and get physically to you. So please read on.
Start registering in an Excel spreadsheet the number of calls and duration. Build a paper trail of hours spent in a week and in a month, and make it a documented case that you need a junior/help desk person to handle those calls and go to people desks. Be prepared to present a case to your superior that it is impossible to do the work you are hired for on the long term: if you spend all your time dealing with calls and putting out fires, you won't gain the knowledge or have the time for improving things for the long term.
It also might help remembering (or reminding your manager) that your cost per hour of work is much higher than a help desk employee. Cost numbers might speak higher for certain type of managers. If you could enlist the help of someone used to do this kind of calculations and extrapolate some numbers about (not) doing maintenance and hiring external consulting teams to do the work you do not have the time to do, so much the better.
Also, establish priorities (with the blessing of your superiors). For instance, the dev department will get a higher priority than the other departments. In my last job I dedicated at least a couple of hours every day to be with the development department, as as a result my job become easier helping them and knowing what they needed in terms of support and infrastructure.
PS. From my experience, in the past, I worked for a corporation that had a business on the side in a industry field that do required a lot of expertise. The previous person on my place was let go twice from two locations, because ultimately, he spent all the time talking with people instead of taking care of issues to guarantee things were automated and working swiftly.
Twice I had to clean up big mistakes after he left. My different take was that I enforced proper channels and escalation, and indeed also had a meeting with the General Manager where I conveyed that they/I had to have another IT person taking care of the corporate desktop PC side for me to focus on our multi-million business side. They got me such person in a few weeks.
Fortunately in my last job in a smallish organization, my superior who hired me knew my talents, and I was never put to deal directly with the day-to-day grind, and dealt only with projects and escalation issues, and thus my salary was largely paid doing complex projects tidying up the house and stabilizing the network and associated services that then did not need to be outsourced to expensive consultants.
Finally, 2 solid business cases on point:
We had something like 30+ wifi authentication tickets per day. Instead of me dealing with each case personally, we had a rookie dealing with them, whilst I studied, tested and became the community expert on the equivalent opensource technology. When my new infrastructure setup and procedures were placed on production, we went from 30 tickets per day to 1.5 tickets per day. Had we not a rookie and an help desk dealing with the 30 tickets per day, the old infra-structure would be still on place, or they would have had to outsource a project to Microsoft, paying probably 1 year worth of my salary.
I also handled a big security problem during 3 weeks, that while my colleagues were handling the complaints of interruptions of Internet connectivity, I did a big infra-structure and security investigation that paid probably for 3 or 4 years of my salary had it been outsourced to security consultants.
However, I did all of this and so much more than those projects because I was not all day putting out fires.
