What should one do in the case when I'm a worker who has studied mathematics and physics and boss has not studied those things. Then the boss asks to try to solve one problem by given methods and you know and can prove that the given method won't solve the problem or even that the problem can't be solved at all. The boss won't accept my reasoning why the problem is unsolvable and he is not willing to read an proof or the theory where the situation has been proved to be impossible.
You do the same thing you would do if you thought it might be solvable: you ask for help. Seniors, the internet community, maybe a paid consultant. After a while, if all of those just deliver reports about how it cannot be done, even your boss will be tired of paying yet another consultant to tell him things you told him half a year ago.
However, this is not delivering any value. While your boss is slowly waking up to the fact that what he wants cannot be done, prepare and regularly show him alternatives to his business problem. Chances are, he's not in the business of formulating mathematical proofs. He is in the business of making money. If the problem cannot be solved soundly in mathematical terms, maybe it can be solved in a way that still earns him money? Sometimes "good enough" is all that's needed, although it makes us theoreticians shiver.
Instead of trying to prove that it cannot be done, try to find out what can be done and how you can do it. For example, if your boss asks for an optimal solution to a problem, and you know it cannot be found in any reasonable time, find a way to find a good solution in a reasonable time and present that. (Your boss will likely not be able to prove that it isn't optimal anyway).
Your boss doesn't want you to solve the given problem by a specific method.
In fact, he doesn't necessarily want the problem that he has posed solving at all.
What he does want is to change an unsatisfactory situation that is affecting the business into something more satisfactory. Give him a way to doing that, and he'll forget the specifics of what he actually asked for.
The scenario you described seems like a typical example of an XY problem.
Often, the hardest part of solving an XY problem is discovering what the real problem actually is!
I hope that I am not coming across as glib, considering that none of us has seen either the text of the problem and are not privy to the reasoning why the problem is unsolvable. Here it is:
redefine the problem so that it is solvable while keeping compatibility with the original definition of the problem, thus keeping the needs of the boss met.
solve the problem and refine the solution to optimize the fit to the situation at hand.
When in an argument with the boss, back your argument with documentation. People will have difficulty with accepting your reasoning if your reasoning is not intuitive.
This question already has several good answers so I'm not sure that this will help.
Try approximating a solution with iterative methods. This is what my experience from when I was studying ME in college suggests to do if possible.
If not it would help if you could boil the proofs down to a graph or simply summarize why there is no solution.