Don’t make them read. I’m not sure if this can be applied directly to your current situation. The thing that stands out to me is that you want people to be able to become experts on the problem on the first go at the issue or to keep re-reading until they understand. That’s novel, but as you’re experiencing, unlikely. There probably isn’t enough time to do that.
I think a good thing that you might be able to do is diagram or draw whatever these things are instead of trying to textually describe them and all the intricacies.
I work with a lot of website front end pieces and sometimes those pieces can have a lot of underlying business rules and corner cases. The times that the most defects are generated are when we have to build from a great big word vomit user story that lists a mountain of requirements meaning that even if you read, re-read, and even read again, you’re likely to miss something key.
One project, we had user stories that actually conflicted with the graphical design document. More often than not, what was actually implemented was implemented according to the design doc because that could be understood much more easily and at a glance. It had flow graphs, UI mockups, and expected functionality, but the source of truth was supposed to be the actual user story even though those were way more difficult to read.
So the first thing that I think would help is if you tried to get everyone to whiteboard these processes and graphically documented them instead of writing long, boring, volumes of text.
I started working with library called Rx that deals with plumbing data about in a functional, reactive way. I read documents until I was blue in the face about how the plumbing was supposed to work. However, what really made the whole thing click were the marble diagrams that explained what data was going where, when, and why. These diagrams are part of the core documentation and every operator/function in the library has a diagram that matches the textual explanation of the feature so that you can quickly understand the thing you are trying to get a grasp of even if you’re just skimming.
Take a cue from that and try to find a way to allow for birds eye views to give a good enough vantage point of what you’re working on but also entices reading the detailed description as well. You say that some of the things you’re trying to detail are things like mathematical proofs, I would think that there must be some way to describe these things “in action” or applied such that someone can get a ballpark understanding of the concept and use the long form text description as a way to fully understand what’s going on.
Maybe I just like books with pictures.