I am an experienced IT developer who recently switched to IT consulting. My current project involves fighting horrible software performance on a server farm my client has rented.

My client recently paid a huge amount for an external benchmark of this rented server farm, and the results were fine. Today I found out that the server farm's host was aware of the exact time of the tests, and they Volkswagened the test (i.e. changed the setup during the test to produce more favourable results than in real world usage) by temporarily adding additional resources to the server farm.

As it turns out, my client actually published the testing times to the host. The tests were conducted at night as not to interfere with live users, but the host needed to switch off some automated night-time tasks - otherwise the results would have been altered by long-running synchronisations that do not occur during daytime.

I should ring the alarm bell and let my client know, as I am responsible for them. Unfortunately, I gained that information through a junior developer employed by the server host, who prattled away about it (probably without knowing that these additional resources were meant to be a secret). As he has been very helpful in the past weeks -- in fact, my most trustworthy and valuable contact in the project -- I don't want to cost him his job.

Even if he did not know about the secrecy, by phoning me he acted somewhat against a standing order of using the ticket system. That helped me a lot in achieving my goals - and that brings up the dilemma.

I'm open for any suggestions, both ethical and pragmatic.

If it matters, my client would not file a lawsuit, but certainly would dump that provider, and my client is one of their most important customers.

  • 169
    Can't you run the test again without telling the provider? Then you have data, not deduction.
    – user15729
    Commented Mar 1, 2016 at 23:41
  • Comments are not for extended discussion; this conversation has been moved to chat.
    – Jane S
    Commented Mar 4, 2016 at 0:29
  • 2
    Are there any metrics you can gleam from the test that would point this out independent of your contact? Does the test show how many nodes were running, how much RAM and MIPS were available, etc?
    – Cloud
    Commented Mar 4, 2016 at 2:32
  • 3
    @user2284570 What is very likely felony billing fraud and what is very likely non-criminal cheating on a benchmark test are very different things. Commented Mar 7, 2016 at 0:47
  • 3
    @user2284570 No, it's an automated comment that indicates you voted or flagged to close as a duplicate. It does not indicate that anyone else agreed with it... and as I have sufficient rep to see close votes, I can verify that there are no actual close votes on this question, and both people who have reviewed this flag so far have disagreed with it. Commented Mar 7, 2016 at 1:00

8 Answers 8


What you ought to do is independently verify what you've been told by this junior developer. For one, you don't actually know that what he said is accurate. There could have been a miscommunication somewhere, he could be misunderstanding, maybe he doesn't have the whole picture from a technical standpoint and is passing on incomplete data, etc. As they say, "trust, but verify."

For another reason, verifying what you were told protects your source. No one needs to know that he told you anything, because you can hold up the data as evidence of what happened, and completely omit the fact that you got tipped off by anyone. You haven't provided enough technical details to advise on how you'd be able to verify this, but it doesn't sound like it would be difficult at all for a decent sysadmin or systems engineer to do so.

Of course, there's always the option of simply stating what your source told you, and saying that to prevent retaliation, you'd rather not/can't/won't name your source. But verifying yourself, so that even having a source is beside the point, is the better approach.

  • 37
  • 14
    @GrzegorzOledzki Yeah, not a new idea by any means, but in the OP's case, it has the benefit of actually being legal, rather than a cesspool of perjury, illegal wiretapping and gross unconstitutionality that has the American founding fathers spinning in their graves. :/ Commented Mar 2, 2016 at 9:57
  • 103
    Considering the generally poor treatment of whistleblowers, stating that you have a source alone could be enough reason for others to start digging and figure out who your source is. Best to not mention it at all.
    – Dorus
    Commented Mar 2, 2016 at 15:11
  • 4
    First paragraph is absolutely the way to go. Commented Mar 3, 2016 at 9:19

While your concern for the junior programmer in this instance is admirable, please keep in mind that this is a situation where your legal obligation is to your client. That said ...

My customer would not file a lawsuite, but certainly would dump that provider. We are one of their most important customers.

Since your client would dump the provider and since the provider is being dishonest, why would you do anything but recommend dumping the provider?

Arguably, you don't need to divulge your source. In fact, you could just tell your client that since the provider's performance has not met what they're paying for, then it's time to walk away. If pressed, you can say that since the service provider knew about the testing in advance that you suspect they allocated extra resources for the test that they aren't providing normally.

If you really want to protect the junior programmer and have a slot he could fill, you could hire him yourself, or your client could hire him. This way he's sure to avoid being fired by the service provider. Also, in case the service provider tries to sue your client for breach of contract, you're sure to have him around as a witness.

  • 16
    I recommend ditching that last paragraph. Despite the disclaimer, it certainly comes off as legal advice, and it doesn't seem practical based on the info OP gave. I think your answer is just fine without it. Commented Mar 2, 2016 at 20:59
  • 3
    The junior has a legal obligation to the court under pain of perjury to tell the truth, the whole truth, and nothing but the truth. If you're going to give legal advice be accurate, otherwise omit it.
    – user207421
    Commented Mar 3, 2016 at 2:45
  • 4
    @EJP: Even with the legal obligation, perjury happens every day. If the junior programmer is told to lie or risk losing his job, you don't know which way he or shee will go.
    – GreenMatt
    Commented Mar 3, 2016 at 15:51
  • 1
    I like this answer because it is action-based. The action is your client dumping the provider. Thus you only need to explain to your client the situation.
    – Brian Risk
    Commented Mar 4, 2016 at 20:15
  • @user1717828: Okay, the last paragraph was been re-worked. Better?
    – GreenMatt
    Commented Mar 8, 2016 at 14:47

A bit to this

From what you know the provider added servers to the farm during the testing. If your customer paid a huge amount for the benchmark then you should have hoped the company hired to perform the benchmark would have picked that up.

Does the benchmark flat exceed the theoretical output of the farm? Can you show the results are just plain not realistic?

Can you somehow run some independent benchmarks that bring the doctored benchmark into question?

Does the provider not have some real time monitoring of basic performance metrics like CPU, IO, bandwidth ...? You should have run some of those during the disclosed benchmark. I know hindsight is 20/20 but still.

You are not looking for the problem. You know the problem and now you just need to prove it. Your source did you a great service by exposing the problem. Can you prove the problem without exposing the source?

Maybe just go with:

These benchmarks are just not matching up with the performance of the application. Either the app is putting out a bigger load than I expect, the servers are not putting out that performance, or I am just plain missing something. Here are items to explore... And in the list of items to explore have a surprise benchmark.

Outing the person that provided the information is a tough one. He would likely get fired. And likely fired in a very bad way in that hard to find another job. I would look hard to find a way to expose this without exposing the person. Personally I would not out them. Without an independent verification it would not likely not hold up anyway.


If the performance in real life does not match the performance during test (and you have the data to prove it), the solution should be easy: notify the provider, and if the performance does not improve, cancel the contract.

You don't have to tell how you found that the test was rigged, only that daily performance does not match the test benchmark (you can even "innocently" ask: did some hardware/software configuration change since the benchmark?). But your lawyers might be interested in talking to the company you paid for the expensive benchmark - was it the same as the service provider, or different?

Regarding your leaker: do not disclose his name, that would be throwing him under a bus for no good reason and no benefit to your own company. But you may want to stay in contact and possibly return the favor: warn him about the need to look for a different job ahead of the crash (when cancellation of the contract is public information, not before). I understand the dilemma that you cannot provide him a good reference (or maybe you can - saying he was helpful and would go the extra mile to communicate with the customer).

The easiest way of returning the favor would be to take him for a coffee sometime (outside of the premises of both companies, and tell him that the meeting is strictly off the record). It could be shortly after cancellation of the project became public, not before, and explain to him what kind of corporate politics he got entangled in, and what complicated ethical dilemma is there free for the taking (for both of you). So he will be better able to deal with it next time. He seems to be an honest and helpful guy, possibly not clued in yet on the intricacies of life - and you have an excellent chance to help to clue him in; help him learn the lesson without damage to his career.

Not sure how he will react next time: not telling you, or not cheating customers? :-)

  • 1
    "when cancellation of contract will be public info, not before" Out of curiosity, why can't he be warned before?
    – anon
    Commented Mar 4, 2016 at 17:25
  • 1
    @QPaysTaxes I don't know what Peter Masiar's reasoning was but I would suggest that if it isn't public knowledge then the OP is probably bound legally to keep quiet about matters between him and his client and not to disclose any information that is not public. If the OP were to tell developer before, he would be in a similar position to the developer. Since the developer leaked this info to the OP then there is a possibility that he may inadvertently leak the info from the OP. Commented Mar 6, 2016 at 11:10
  • @QPaysTaxes - pedro_sland got it right Commented Mar 7, 2016 at 14:38

The other answers are really good, but I didn't see this explicitly stated in any of them -

Why wouldn't your recommendation just be that their external benchmarking vendor perform another test as their testing method was flawed. If you know that the information was published and services were turned off, I'm sure you can find a NIST or ISO standard stating proper procedure for this type of testing. You could even seek best practices information from ISACA.

Doesn't matter what you know, just that by publishing the time and the environment being altered, it's not a valid test. Even then, I might go to the external testing company and ask them what they were thinking.

Sounds like both the bench marking company and the provider were involved in creating a situation for inaccurate results and were IMO being at least negligent and at most collusive.

Again, I don't know what job you were hired for, but if any of this is within scope, it's a good way to avoid divulging any confidential information.

  • Test is flawed - This is what I was thinking.
    – Aaron Hall
    Commented Mar 2, 2016 at 14:53
  • 2
    I don't see how you can pin blame on the external benchmarking group when the question specifies that his client was the one who divulged the testing times.
    – pydsigner
    Commented Mar 2, 2016 at 18:51
  • @pydsigner yeah, you're right, I misread. I guess that would exclude the collusive possibility, but still negligence is possible. I'd go to the client and ask what the hell they were thinking instead of the external group. Commented Mar 2, 2016 at 19:07

Disclose everything to your client and tell him about the information you've got. You have a professional obligation to give him the best service possible and you have no professional obligation towards the data center's junior developer. Your client literally paid you to discover things like this.

After you have disclosed the information you have acquired (which your client is paying you for). Discuss verifying what the junior developer has said.

If it checks out - definitely work with another provider.

As people, we try to be nice and fair and that's admirable. As a professional you have an obligation to provide the best service possible to your client.

  • 3
    This seems like a really bad idea to do without first doing what HopelessNoob has said.
    – enderland
    Commented Mar 2, 2016 at 13:10
  • 11
    Professional or not, you may have a moral obligation to the junior developer who risks facing huge problems in his life for at best being a good whistleblower (and meeting his professional obligation!) or at worst making a fairly innocent comment.
    – komodosp
    Commented Mar 2, 2016 at 14:59
  • 2
    You certainly do have an obligation to provide the best possible service to your client, but I don't see how outing the junior dev makes that service appreciably better. You certainly don't want to get known as the guy one cannot innocently and honestly talk with without risking one's job.
    – Law29
    Commented Mar 3, 2016 at 21:16

The plain fact that the actual application has "horrible software performance", while in tests "the results were fine" should ring a bell for anyone and can be used as a starting point for discussion, or challenging the tests. You could/ should raise suspicion on the test farm with the client on this discrepancy, specially given that you are "responsible for them".

Additionally, the original issue at hand remains unresolved and open for further analysis, with or without involving the hugely paid external company. Tools are available for many technologies to monitor production performance without really impacting it; you might want to use that to quantify "horrible". Adding on to HopelessN00b's reply, this would be another way of verifying.


Your source is a "junior developer employed by the server host, who prattled away about it". In other words, as far as you are concerned, you did absolutely nothing wrong getting that information.

Your duty is not to the ISP, or the junior developer at the ISP, but to your client. So I believe that you should inform your client. You might mention what kind of source you had without mentioning the actual source. What will happen: Your client will stop using an ISP that is cheating, the ISP will lose the client they are cheating, and the junior developer will hopefully not talk any more about this are will be in danger.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .