The quality of your portfolio site will affect the perceived quality of your work
Regardless of whether you are a physics engineer, back-end programmer or any other discipline (even those unrelated to development) - the quality of anything you show to a potential employer will influence their opinion of you.
This is obviously different from the way it will influence their opinion if you were actively seeking a design or front-end role. That is, you do not need to have a site with all the bells and whistles, or otherwise outstanding. But the quality will affect how the word within it is percieved.
As a back-end programmer, I'd recommend either:
Using a templated site that is clearly a template, but makes your work clear and easy to assess.
Give direct access to a source repository that holds your projects.
Although counter intuitive, in my experience - if you cannot create a solid website yourself, you are far better to make this clear (by choosing a template that is visually obvious as a template). This way, you separate the mental judgement of your personal work/skills from judgement of the site you've used to hold it. The more simple, and clear a template you can find, the better.
Taking that to the extreme, it can also be beneficial to just present your work on a web-accessible source control platform (such as Github). By doing this, you remove yourself even further from the judgement of the host site - it's very clear what is your work, and what isn't.
On an additional note, using a site like Github also enforces the fact you understand these tools and are comfortable working in a technical-facing environment. If you do choose to do this, ensure your work is well organised and easy for a non-technical person to download.