I have done freelancing as a type of self-employment before, so I could offer an angle for you to look at.
As a self employed person, your income is not stable. So it's important to evaluate what the bottom line is for you. Since income is not guaranteed round the year, freelancer's charges are even more than an employee's would be(per hour). The fact that you are willing to actually work below what someone else on the market is prepared to offer could later be a cause of concern for you.
I feel some sort of loyalty to the client and I don’t want to lose what I think is good work relationship with him
Maintaining a good relationship is the client's responsibility too - it's only fair that if you underpay someone, sooner or later they leave looking for better opportunities. Don't just look at the 'relationship', look at what's best for you.
Also, evaluate the company you're working with.
Loyalty to the client of the company that I'm working with directly and not that much to the company itself.The client has no influence on how much they pay me, so it's kind of not his fault
If the client wants to pay you higher but the company's not allowing him to, is the company really somebody you'd want to do business with? Move on to some better company(bonus if the client will be able to re-contract with you through that other company).