Over the last ~10 years I have worked as a software developer, then a mathematical modeller, then back to software developer.
Now, it is obviously important how much time out of the ~10 years you were a mathematical modeller. Seven years is one thing, three months is another. Even if I went to the extremes a bit, you get my point. If the time you spent actually being a mathematical modeller was closer to the "irrelevant" side, then you need to analyze your options from that point of view. Everything is NOT lost, just a bit more difficult.
I know my CV looks terrible, but is there anything I can do to improve my chances?
Well, that is the excellent attitude to make you undesirable for pretty much any job. You do not even need to say that during the interview. Just think it. Just store it in the hidden parts of your memory. The interviewer (especially if they are experienced) will read that low self-evaluation, and treat you accordingly.
Example from my own life: my experience is mostly about software engineering for automotive. I have done may things around that subject, there was a long time. However, once I wanted to switch jobs, and I found a position as a software "guy" to work (employed) on improving and maintaining some open-source compiler for some processor, under Linux. Obviously, a job very far from my experience. I went to the interview knowing very well my value and my abilities, and my willingness to get involved and learn and improve. When I knew some answer, I went ahead and provided it. Because my specific experience was practically zero, I did not know several answers, and I openly admitted to that. At the end, I pretty much got the job. I did not accept it, simply because I moved to another job, more compatible with my personal needs and priorities at that time.
So, the conclusion: know yourself, do not let your limits define who you are. Redefine yourself, push your limits. "Boldly go where no-one has gone before!"
On a couple of occasions they have asked if would like to re-interview for a back-end software developer type position. I politely decline.
That is another thing you can improve. Do not let opportunities fly by you. There are not many companies who need a mathematical modeller. If such a company wants to hire you, then find a way to turn the situation in your favor. Try an approach along these lines:
Being a (back-end developer) is not my first priority. However, I am willing to make a gradual transition, in a way that covers both our needs, to end-up with a win-win situation. I propose that I start doing (80% ?) a back-end job, and (20% ?) mathematical modelling. We evaluate my performance as a mathematical modeller in (one ?) year, and if it is satisfactory, then we review my work allocation, targeting 100% work as a mathematical modeller.
This approach gives you the huge benefit of accumulating experience as a mathematical modeller - something that is in your target. Even if this job is not very good, the next one might be a lot better - you will start from a different (better) starting point.
Most of the time I don't get an interview and when I do I'm rejected.
Again, you concentrate on the failure. What you need to see from this is that actually you were accepted for an interview. That is already a big win. It means that the company already sees potential in you. You just need to convince them.
I will not repeat the good advice from @Old_Lamplighter. But I will give you another example which actually helped me in my personal life (although not really in the direct way). Watch the movie "Hitch" (2005) with Will Smith and understand all the advice "The Doctor" gives to his customers. Once "she" said "yes", you move forward. Do not make efforts to improve a "yes". Of course, you will understand that in your case the love (or the woman) is actually the job you are targeting. Not really everything will apply ;)
have a PhD in STEM subject
While some companies are happy to have people with degrees, many companies treat such people with a cold shoulder. Search for the subject of "over-qualification". When I first heard about it, I thought it was a joke, but in time I understood that it was actually a thing.
What you might want to try is to not advertise your PhD. See what happens. If they require a PhD and they are on the verge to reject you for not having the title, just then show them that you have the needed qualification.
If asked why you kept the information hidden, just say that you did not keep it hidden. You just did not consider the title very important for the job - and admit that you were a bit wrong, using a smile.
Sometimes it will help if you study the company before the interview. See what they do, what they advertise. See where you might have a contribution. If they have people which appeared in the media or in scientific journals, study what they did. Analyze yourself and see if you fit in their projects. See where they are stuck with their work (this information is often written in plain words, both in the media, and in the scientific articles). If you have an idea to help, contact these people and see if they are willing to listen. If yes, your chances for a job are dramatically increased.
Other than these, I wish you good luck and success.