Is it professional to thank a university student for a very well-written homework when correcting the homework? I am thinking of something like writing "very nice" beside of the total mark on the homework paper.
I don't see why it wouldn't be
At least in North America, that is quite a normal thing for a teacher to do. I received many such "thank yous" when I was in school. Not sure where you are located, so it may differ.
You don't mention the age of your students.
For younger kids, my teacher used to stick coloured stars to our homework, with silver or gold for really good work.
In secondary school, "good work" or similar written on returned homework was the norm.
I never got any praise at university, but I may not have produced any good work :-/
So, yes, that seems fine and will hopefully motivate your students, which can only be a good thing.