The Workplace Stack Exchange is a question and answer site for members of the workforce navigating the professional setting. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am an experienced IT developer who recently switched to IT consulting. My current project involves fighting horrible software performance on a server farm my client has rented.

My client recently paid a huge amount for an external benchmark of this rented server farm, and the results were fine. Today I found out that the server farm's host was aware of the exact time of the tests, and they Volkswagened the test (i.e. changed the setup during the test to produce more favourable results than in real world usage) by temporarily adding additional resources to the server farm.

As it turns out, my client actually published the testing times to the host. The tests were conducted at night as not to interfere with live users, but the host needed to switch off some automated night-time tasks - otherwise the results would have been altered by long-running synchronisations that do not occur during daytime.

I should ring the alarm bell and let my client know, as I am responsible for them. Unfortunately, I gained that information through a junior developer employed by the server host, who prattled away about it (probably without knowing that these additional resources were meant to be a secret). As he has been very helpful in the past weeks -- in fact, my most trustworthy and valuable contact in the project -- I don't want to cost him his job.

Even if he did not know about the secrecy, by phoning me he acted somewhat against a standing order of using the ticket system. That helped me a lot in achieving my goals - and that brings up the dilemma.

I'm open for any suggestions, both ethical and pragmatic.

If it matters, my client would not file a lawsuit, but certainly would dump that provider, and my client is one of their most important customers.

share|improve this question
120  
Can't you run the test again without telling the provider? Then you have data, not deduction. – fredsbend Mar 1 at 23:41
    
Comments are not for extended discussion; this conversation has been moved to chat. – Jane S Mar 4 at 0:29
    
Any chance of some clarity about what "adding additional resources" means? What "resources"? – user2338816 Mar 4 at 1:26
1  
Are there any metrics you can gleam from the test that would point this out independent of your contact? Does the test show how many nodes were running, how much RAM and MIPS were available, etc? – DevNull Mar 4 at 2:32
    

What you ought to do is independently verify what you've been told by this junior developer. For one, you don't actually know that what he said is accurate. There could have been a miscommunication somewhere, he could be misunderstanding, maybe he doesn't have the whole picture from a technical standpoint and is passing on incomplete data, etc. As they say, "trust, but verify."

For another reason, verifying what you were told protects your source. No one needs to know that he told you anything, because you can hold up the data as evidence of what happened, and completely omit the fact that you got tipped off by anyone. You haven't provided enough technical details to advise on how you'd be able to verify this, but it doesn't sound like it would be difficult at all for a decent sysadmin or systems engineer to do so.

Of course, there's always the option of simply stating what your source told you, and saying that to prevent retaliation, you'd rather not/can't/won't name your source. But verifying yourself, so that even having a source is beside the point, is the better approach.

share|improve this answer
29  
12  
@GrzegorzOledzki Yeah, not a new idea by any means, but in the OP's case, it has the benefit of actually being legal, rather than a cesspool of perjury, illegal wiretapping and gross unconstitutionality that has the American founding fathers spinning in their graves. :/ – HopelessN00b Mar 2 at 9:57
84  
Considering the generally poor treatment of whistleblowers, stating that you have a source alone could be enough reason for others to start digging and figure out who your source is. Best to not mention it at all. – Dorus Mar 2 at 15:11
4  
First paragraph is absolutely the way to go. – T.J. Crowder Mar 3 at 9:19

While your concern for the junior programmer in this instance is admirable, please keep in mind that this is a situation where your legal obligation is to your client. That said ...

My customer would not file a lawsuite, but certainly would dump that provider. We are one of their most important customers.

Since your client would dump the provider and since the provider is being dishonest, why would you do anything but recommend dumping the provider?

Arguably, you don't need to divulge your source. In fact, you could just tell your client that since the provider's performance has not met what they're paying for, then it's time to walk away. If pressed, you can say that since the service provider knew about the testing in advance that you suspect they allocated extra resources for the test that they aren't providing normally.

If you really want to protect the junior programmer and have a slot he could fill, you could hire him yourself, or your client could hire him. This way he's sure to avoid being fired by the service provider. Also, in case the service provider tries to sue your client for breach of contract, you're sure to have him around as a witness.

share|improve this answer
13  
I recommend ditching that last paragraph. Despite the disclaimer, it certainly comes off as legal advice, and it doesn't seem practical based on the info OP gave. I think your answer is just fine without it. – user1717828 Mar 2 at 20:59
3  
The junior has a legal obligation to the court under pain of perjury to tell the truth, the whole truth, and nothing but the truth. If you're going to give legal advice be accurate, otherwise omit it. – EJP Mar 3 at 2:45
4  
@EJP: Even with the legal obligation, perjury happens every day. If the junior programmer is told to lie or risk losing his job, you don't know which way he or shee will go. – GreenMatt Mar 3 at 15:51
1  
I like this answer because it is action-based. The action is your client dumping the provider. Thus you only need to explain to your client the situation. – Brian Risk Mar 4 at 20:15
    
@user1717828: Okay, the last paragraph was been re-worked. Better? – GreenMatt Mar 8 at 14:47

A bit to this

From what you know the provider added servers to the farm during the testing. If your customer paid a huge amount for the benchmark then you should have hoped the company hired to perform the benchmark would have picked that up.

Does the benchmark flat exceed the theoretical output of the farm? Can you show the results are just plain not realistic?

Can you somehow run some independent benchmarks that bring the doctored benchmark into question?

Does the provider not have some real time monitoring of basic performance metrics like CPU, IO, bandwidth ...? You should have run some of those during the disclosed benchmark. I know hindsight is 20/20 but still.

You are not looking for the problem. You know the problem and now you just need to prove it. Your source did you a great service by exposing the problem. Can you prove the problem without exposing the source?

Maybe just go with:

These benchmarks are just not matching up with the performance of the application. Either the app is putting out a bigger load than I expect, the servers are not putting out that performance, or I am just plain missing something. Here are items to explore... And in the list of items to explore have a surprise benchmark.

Outing the person that provided the information is a tough one. He would likely get fired. And likely fired in a very bad way in that hard to find another job. I would look hard to find a way to expose this without exposing the person. Personally I would not out them. Without an independent verification it would not likely not hold up anyway.

share|improve this answer

If the performance in real life does not match the performance during test (and you have the data to prove it), the solution should be easy: Notify the provider, and if the performance does not improve, cancel the contract.

You don't have to tell how you found that test was rigged, only that daily performance does not match the test benchmark (you can even "innocently" ask: did some hardware/software configuration changed since the benchmark?). But your lawyers might be interested to talk to the company you paid for the expensive benchmark - was it the same as service provider, or different?

Regarding your leaker: do not disclose his name, that would be throwing him under the bus for no good reason and no benefit to your own company. But you may want to stay in contact and possibly return the favor: warn him about the need to look for different job ahead of the crash (when cancellation of the contract will be public information, not before). I understand the dilemma that you cannot provide him a good reference (or maybe you can - saying he was helpful and go the extra mile to communicate with the customer).

The easiest way of returning the favor would be to take him for a coffee sometimes (outside of premises of both companies, and tell him that meeting is strictly off the record). It could be shortly after cancellation of the project became public, not before, and explain to him what kind of corporate politics he got entangled in, and what complicated ethical dilemma is there free for the taking (for both of you). So he will be better able to deal with it next time. He seems to be an honest and helpful guy, possibly not clued in yet on the intricacies of life - and you have excellent chance to help to clue him in, help him learning the lesson without damage to his career.

Not sure how he will react next time: not telling you, or not cheating customers? :-)

share|improve this answer
1  
"when cancellation of contract will be public info, not before" Out of curiosity, why can't he be warned before? – QPaysTaxes Mar 4 at 17:25
1  
@QPaysTaxes I don't know what Peter Masiar's reasoning was but I would suggest that if it isn't public knowledge then the OP is probably bound legally to keep quiet about matters between him and his client and not to disclose any information that is not public. If the OP were to tell developer before, he would be in a similar position to the developer. Since the developer leaked this info to the OP then there is a possibility that he may inadvertently leak the info from the OP. – pedro_sland Mar 6 at 11:10
    
@QPaysTaxes - pedro_sland got it right – Peter Masiar Mar 7 at 14:38

The other answers are really good, but I didn't see this explicitly stated in any of them -

Why wouldn't your recommendation just be that their external benchmarking vendor perform another test as their testing method was flawed. If you know that the information was published and services were turned off, I'm sure you can find a NIST or ISO standard stating proper procedure for this type of testing. You could even seek best practices information from ISACA.

Doesn't matter what you know, just that by publishing the time and the environment being altered, it's not a valid test. Even then, I might go to the external testing company and ask them what they were thinking.

Sounds like both the bench marking company and the provider were involved in creating a situation for inaccurate results and were IMO being at least negligent and at most collusive.

Again, I don't know what job you were hired for, but if any of this is within scope, it's a good way to avoid divulging any confidential information.

share|improve this answer
    
Test is flawed - This is what I was thinking. – Aaron Hall Mar 2 at 14:53
2  
I don't see how you can pin blame on the external benchmarking group when the question specifies that his client was the one who divulged the testing times. – pydsigner Mar 2 at 18:51
    
@pydsigner yeah, you're right, I misread. I guess that would exclude the collusive possibility, but still negligence is possible. I'd go to the client and ask what the hell they were thinking instead of the external group. – Raystafarian Mar 2 at 19:07

Disclose everything to your client and tell him about the information you've got. You have a professional obligation to give him the best service possible and you have no professional obligation towards the data center's junior developer. Your client literally paid you to discover things like this.

After you have disclosed the information you have acquired (which your client is paying you for). Discuss verifying what the junior developer has said.

If it checks out - definitely work with another provider.

As people, we try to be nice and fair and that's admirable. As a professional you have an obligation to provide the best service possible to your client.

share|improve this answer
3  
This seems like a really bad idea to do without first doing what HopelessNoob has said. – enderland Mar 2 at 13:10
10  
Professional or not, you may have a moral obligation to the junior developer who risks facing huge problems in his life for at best being a good whistleblower (and meeting his professional obligation!) or at worst making a fairly innocent comment. – colmde Mar 2 at 14:59
2  
You certainly do have an obligation to provide the best possible service to your client, but I don't see how outing the junior dev makes that service appreciably better. You certainly don't want to get known as the guy one cannot innocently and honestly talk with without risking one's job. – Law29 Mar 3 at 21:16

The plain fact that the actual application has "horrible software performance", while in tests "the results were fine" should ring a bell for anyone and can be used as a starting point for discussion, or challenging the tests. You could/ should raise suspicion on the test farm with the client on this discrepancy, specially given that you are "responsible for them".

Additionally, the original issue at hand remains unresolved and open for further analysis, with or without involving the hugely paid external company. Tools are available for many technologies to monitor production performance without really impacting it; you might want to use that to quantify "horrible". Adding on to HopelessN00b's reply, this would be another way of verifying.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.