as I cannot see anybody being harmed this way
I'll start to answer the question from here.
Your current company is making profit by putting you work for the the client. They pay you X dollars per day. They charge the client X + P dollars per day where P is the profit your company is making.
Now you quit your job. Your company will have to find replacement. They cannot charge the client any money between the time you quit and the replacement starts to work for the client. Suppose it takes 5 days for them to find the replacement and put that replacement to work, your company lose 5 x P dollars. So, how can you say nobody being harmed?
Now, let's look at your side. The client will hire you after you quit your company because they know you can perform well on the job. How do they know that? Because your company put you work there in the first place so that you had the opportunity to show your performance. If your company never put you work there, how would the client know you would perform well?
To answer your question, is it ethical? No. At least, it's not perfectly ethical to me. But, there are just too many un-ethical things in the business world. Your case is not a big deal. If I were the manager of your company, I would be upset. If I were your client, I would be happy because your coming to work for me saves me a lot time and money to interview people and find a good worker. So, somebody gets hurt and somebody else gets help. If you want me to say your act is ethical, sorry, I would never say that. You just did yourself the best of your interests. That's all.
P.S. Please note that I assume you'll have no legal issue when you go to work for your client. If there are legal issues, all bets are off and I am not a lawyer.