Provider is "cheating", but I don't want to disclose how I found out

Question

I am an experienced IT developer who recently switched to IT consulting. My current project involves fighting horrible software performance on a server farm my client has rented.

My client recently paid a huge amount for an external benchmark of this rented server farm, and the results were fine. Today I found out that the server farm's host was aware of the exact time of the tests, and they Volkswagened the test (i.e. changed the setup during the test to produce more favourable results than in real world usage) by temporarily adding additional resources to the server farm.

As it turns out, my client actually published the testing times to the host. The tests were conducted at night as not to interfere with live users, but the host needed to switch off some automated night-time tasks - otherwise the results would have been altered by long-running synchronisations that do not occur during daytime.

I should ring the alarm bell and let my client know, as I am responsible for them. Unfortunately, I gained that information through a junior developer employed by the server host, who prattled away about it (probably without knowing that these additional resources were meant to be a secret). As he has been very helpful in the past weeks -- in fact, my most trustworthy and valuable contact in the project -- I don't want to cost him his job.

Even if he did not know about the secrecy, by phoning me he acted somewhat against a standing order of using the ticket system. That helped me a lot in achieving my goals - and that brings up the dilemma.

I'm open for any suggestions, both ethical and pragmatic.

If it matters, my client would not file a lawsuit, but certainly would dump that provider, and my client is one of their most important customers.

Answer 1

What you ought to do is independently verify what you've been told by this junior developer. For one, you don't actually know that what he said is accurate. There could have been a miscommunication somewhere, he could be misunderstanding, maybe he doesn't have the whole picture from a technical standpoint and is passing on incomplete data, etc. As they say, "trust, but verify."

For another reason, verifying what you were told protects your source. No one needs to know that he told you anything, because you can hold up the data as evidence of what happened, and completely omit the fact that you got tipped off by anyone. You haven't provided enough technical details to advise on how you'd be able to verify this, but it doesn't sound like it would be difficult at all for a decent sysadmin or systems engineer to do so.

Of course, there's always the option of simply stating what your source told you, and saying that to prevent retaliation, you'd rather not/can't/won't name your source. But verifying yourself, so that even having a source is beside the point, is the better approach.

Answer 2

While your concern for the junior programmer in this instance is admirable, please keep in mind that this is a situation where your legal obligation is to your client. That said ...

My customer would not file a lawsuite, but certainly would dump that provider. We are one of their most important customers.

Since your client would dump the provider and since the provider is being dishonest, why would you do anything but recommend dumping the provider?

Arguably, you don't need to divulge your source. In fact, you could just tell your client that since the provider's performance has not met what they're paying for, then it's time to walk away. If pressed, you can say that since the service provider knew about the testing in advance that you suspect they allocated extra resources for the test that they aren't providing normally.

If you really want to protect the junior programmer and have a slot he could fill, you could hire him yourself, or your client could hire him. This way he's sure to avoid being fired by the service provider. Also, in case the service provider tries to sue your client for breach of contract, you're sure to have him around as a witness.

Answer 3

A bit to this

From what you know the provider added servers to the farm during the testing. If your customer paid a huge amount for the benchmark then you should have hoped the company hired to perform the benchmark would have picked that up.

Does the benchmark flat exceed the theoretical output of the farm? Can you show the results are just plain not realistic?

Can you somehow run some independent benchmarks that bring the doctored benchmark into question?

Does the provider not have some real time monitoring of basic performance metrics like CPU, IO, bandwidth ...? You should have run some of those during the disclosed benchmark. I know hindsight is 20/20 but still.

You are not looking for the problem. You know the problem and now you just need to prove it. Your source did you a great service by exposing the problem. Can you prove the problem without exposing the source?

Maybe just go with:

These benchmarks are just not matching up with the performance of the application. Either the app is putting out a bigger load than I expect, the servers are not putting out that performance, or I am just plain missing something. Here are items to explore... And in the list of items to explore have a surprise benchmark.

Outing the person that provided the information is a tough one. He would likely get fired. And likely fired in a very bad way in that hard to find another job. I would look hard to find a way to expose this without exposing the person. Personally I would not out them. Without an independent verification it would not likely not hold up anyway.

Answer 4

If the performance in real life does not match the performance during test (and you have the data to prove it), the solution should be easy: notify the provider, and if the performance does not improve, cancel the contract.

You don't have to tell how you found that the test was rigged, only that daily performance does not match the test benchmark (you can even "innocently" ask: did some hardware/software configuration change since the benchmark?). But your lawyers might be interested in talking to the company you paid for the expensive benchmark - was it the same as the service provider, or different?

Regarding your leaker: do not disclose his name, that would be throwing him under a bus for no good reason and no benefit to your own company. But you may want to stay in contact and possibly return the favor: warn him about the need to look for a different job ahead of the crash (when cancellation of the contract is public information, not before). I understand the dilemma that you cannot provide him a good reference (or maybe you can - saying he was helpful and would go the extra mile to communicate with the customer).

The easiest way of returning the favor would be to take him for a coffee sometime (outside of the premises of both companies, and tell him that the meeting is strictly off the record). It could be shortly after cancellation of the project became public, not before, and explain to him what kind of corporate politics he got entangled in, and what complicated ethical dilemma is there free for the taking (for both of you). So he will be better able to deal with it next time. He seems to be an honest and helpful guy, possibly not clued in yet on the intricacies of life - and you have an excellent chance to help to clue him in; help him learn the lesson without damage to his career.

Not sure how he will react next time: not telling you, or not cheating customers? :-)

Answer 5

The other answers are really good, but I didn't see this explicitly stated in any of them -

Why wouldn't your recommendation just be that their external benchmarking vendor perform another test as their testing method was flawed. If you know that the information was published and services were turned off, I'm sure you can find a NIST or ISO standard stating proper procedure for this type of testing. You could even seek best practices information from ISACA.

Doesn't matter what you know, just that by publishing the time and the environment being altered, it's not a valid test. Even then, I might go to the external testing company and ask them what they were thinking.

Sounds like both the bench marking company and the provider were involved in creating a situation for inaccurate results and were IMO being at least negligent and at most collusive.

Again, I don't know what job you were hired for, but if any of this is within scope, it's a good way to avoid divulging any confidential information.

Answer 6

Disclose everything to your client and tell him about the information you've got. You have a professional obligation to give him the best service possible and you have no professional obligation towards the data center's junior developer. Your client literally paid you to discover things like this.

After you have disclosed the information you have acquired (which your client is paying you for). Discuss verifying what the junior developer has said.

If it checks out - definitely work with another provider.

As people, we try to be nice and fair and that's admirable. As a professional you have an obligation to provide the best service possible to your client.

Answer 7

The plain fact that the actual application has "horrible software performance", while in tests "the results were fine" should ring a bell for anyone and can be used as a starting point for discussion, or challenging the tests. You could/ should raise suspicion on the test farm with the client on this discrepancy, specially given that you are "responsible for them".

Additionally, the original issue at hand remains unresolved and open for further analysis, with or without involving the hugely paid external company. Tools are available for many technologies to monitor production performance without really impacting it; you might want to use that to quantify "horrible". Adding on to HopelessN00b's reply, this would be another way of verifying.

Answer 8

Your source is a "junior developer employed by the server host, who prattled away about it". In other words, as far as you are concerned, you did absolutely nothing wrong getting that information.

Your duty is not to the ISP, or the junior developer at the ISP, but to your client. So I believe that you should inform your client. You might mention what kind of source you had without mentioning the actual source. What will happen: Your client will stop using an ISP that is cheating, the ISP will lose the client they are cheating, and the junior developer will hopefully not talk any more about this are will be in danger.