The Core Strategy:
It sounds like you are willing to go above and beyond for your friend Brian, but with some limits. The situation is souring, so the best thing you can do for yourself and for your relationship with Brian is to get out of the position of Derek having further expectations of you.
You painted the scenario of yourself being a middle man (since you're paying another hosting provider). So, enjoy the benefits of being a middle man. If,, at any time, you don't find yourself being a middle man, and you're the end resource that is obligated to provide a solution, but you're finding yourself unable to produce a solution yourself, then the solution you provide may be to transform yourself into being a middle man who recommends using another resource to reach a desirable solution. Using this strategy, you can always have an available solution in your back pocket.
This should eventually lead to a scenario where some technical expert manages to accomplish the desired goal, or a scenario where a possible solution appears to have a cost greater than what Derek desires. In the latter (less desirable) case, at least nobody will be saying that you're the incompetent person who only provided shut doors. (Instead, you offered a solution, which was simply not desirable enough. Maybe not good, but not as bad as not having a usable solution.)
This answer is a bit lengthy because I do have some experienced thoughts on how this scenario can play out rather positively for everyone. So I'm sharing my thoughts. None of the following is meant to alter the shorter core answer that I provided above.
If the issue is actually with the hosting, you may be able to get support. After all, you're a customer in this scenario. You did say "I set up a reseller account with a large hosting provider."
So, as the customer, try tapping their support and see what the reseller does.
See if you can get them to resolve the situation. If so, great! If not, find out why not, and now you know an excuse/reason that a professional company uses to decline to perform that work. Then you will know know of a way that a hosting company can respond. You can decide whether that excuse is satisfying for you to be able to say. If not, consider how you'd like to polish that up, but at least then you're not needing to start your brainstorming from scratch. (Being a middle person can be a position with some advantages.)
Here is what you can then say to Derek (again, if needed) -
"Internal investigations has revealed no identified problems. I reviewed the situation and my findings are this...."
If there is any additional evidence that the issue is related to hosting, please let me know. Otherwise, this may need to be further reviewed by the designer (Brian) or investigated/handled by someone performing an IT role."
See, there are at least three possible roles here.
- Web designer
- IT/Networking expert
- ISP Host
All three of these roles may be somewhat performed, loosely, by the alliance (currently consisting of you and Brian). However, if your non-expert attempts to resolve the situation aren't finding the issue, rope in more expertise.
It is problematic that the description of the issue is "his site was running too slowly." Determine tasks that are being performed, and precise numbers. Such numbers will need to be met in order to identify any cost for an improved situation.
If you can't find a technical cause for a problem, the next best approach may be to make Darek's requests for your free support go away, by letting him know how much a solution will cost. (And don't quote him final prices, as you don't want to be responsibly on the hook for an overall solution involving unknown territory.)
"It looks like this is the level of performance available with this solution. The other option I am seeing for improved performance involves getting this superior hardware, and/or roping in this other resource to provide additional specialized expertise."
You don't want to slam any doors closed, but what you can do is to show doors that are open when enough additional cash is provided. Then Derek can't ever say, "Charlie was the cause of a problem, with no available solution." All Derek can say is "Charlie wasn't able to deliver what I was wanting withing the budget of what I was wanting." The latter is far less awful.
And, if Derek does infuse the situation with more money, then you have another organization that is providing Derek with more resources for a solution. Let them handle the area of their expertise (or, if the situation grows beyond their control, let them figure out what to recommend next).
This is a key reason to make sure you're not promising any specific solution for any specific dollar amount. Eventually Derek will get what he wants, or Derek will hit some sort of financial wall that could have been resolved if more money was dumped into the picture, but he shouldn't ever be told that a resolution can't be reached.
If you try utilizing a provider and they pass the buck, hopefully they at least give you a direction where people can go, so you're not left holding any responsibility without an available solution that can be pursued.
By the time you get enough professionals involved, someone will be able to pinpoint that problem. Everyone in this scenario should be reluctant to toss Brian under the bus, because Brian is a partner with everybody involved in this solution.
But, what if Brian's web design is what is actually at fault, causing the slowness? Well, if there is still sufficient cause for technical experts to say that Brian's design seems to be contributing to the issue, then you can now have an ally point this out to Brian.
Now you don't have to jeopardize a friendship with Brian, because you can explain to Brian, "I didn't initiate any Brian-blaming. That was a conclusion reached by this other company. Please work this out with the other company, or else we can find a different company if you like." If Brian's the person who charges Derek money, then Brian can fix his own issues or Brian can change which technical partners he wishes to utilize. Eventually, if enough companies keep pointing the finger in the same direction (at Brian), he ought to realize that what he's doing is causing fingers to point in his direction. If your finger isn't the one initiating the pointing, Brian won't have much reason to be getting mad at you.
Such squabbling should, of course, happen behind the scenes, without Derek seeing who the responsibility is falling on. So, then, Brian doesn't look bad to Derek. None of that looks particularly bad on anything you're doing.
In the end, you can always provide a solution, even if that involves turning your role into that of being a middleman that gets additional resources involved (at additional cost to the client).