Take the 2-minute tour ×
The Workplace Stack Exchange is a question and answer site for members of the workforce navigating the professional setting. It's 100% free, no registration required.

I've got an ongoing relationship with a long term business partner as a consultant where his role is project manager (task manager + direction), and my role is a contracted developer. He has a tendency to micromanage my time with his tasks and oversight, but also has a strong sense of perfection.

Recently with every single programming task undertaken he is asking me to confirm that I have "100% confidence that this fix will not break any existing features or cause any adverse effects on user experience". If I can't affirm that, he assumes I haven't tested it well enough or should go check it again. And yes, he actually asks this every single bug fix, it isn't just implied.

As a developer, I do test my work on multiple unit cases, but can't say that it is possible to fully regression test the entire product for each 2-hour task I accomplish. There is no QA team either. The product has lots of intermingled parts throughout (not just self-contained pages), some 40,000 lines of code written over 4 years, and sometimes unexpected things happen that we weren't even aware of. I sense he sees this as poor testing.

How should I respond to his question in this case, without seeming incompetent? Honestly I never have 100% confidence sitewide, but I do have confidence in my testing methods. And, as a developer I also know that it isn't uncommon for unexpected bugs to emerge later from these core changes.

EDIT:
I'm not necessarily looking for a solution to make this 100%, as our group doesn't have the time or resources to implement a full QA process or get into setting up automated solutions. I'm looking for how to interact with the manager around existing work, especially when he isn't entirely a technical person himself. He isn't a programmer.

share|improve this question
11  
"but [he] also has a strong sense of perfection." - could you tell us, has he ever made a mistake? –  AakashM Mar 5 at 16:33
1  
The relationship is such that he judges my work and not the reverse. So, while no one is perfect in the grand scheme, he only accounts for my bugs. –  Miro Mar 5 at 16:36
2  
this question rather belongs to Programmers, where it looks like a duplicate of Functional testing before code checkin and Should a developer also act as a tester? –  gnat Mar 5 at 20:05
12  
@gnat: I disagree. This question is about the human interaction in presence of "established" testing rules, rather than a question about how to do testing. –  PlasmaHH Mar 5 at 20:14
2  
@PlasmaHH The answer to this problem is to build automated regression tests... How is that a Workplace navigation solution –  ReallyTiredOfThisGame Mar 5 at 20:16

11 Answers 11

up vote 166 down vote accepted

How should I respond to his question in this case, without seeming incompetent? Honestly I never have 100% confidence sitewide, but I do have confidence in my testing methods. And, as a developer I also know that it isn't uncommon for unexpected bugs to emerge later from these core changes.

The project manager doesn't understand software well enough, and certainly doesn't understand testing well enough. Perhaps he needs to be educated.

If you had a professional QA Department, I'd tell you to enlist the support of the QA Manager in explaining to this Project Manager the nature of software, the nature of bugs, and the nature of testing. I'd have the QA Manager indicate why it is simply not possible to test every condition, and how releasing/not releasing is a business activity aided by the findings from testing, but never by perfect information.

You may wish to get a copy of Gerald Weinberg's excellent book "Perfect Software and other illusions about testing". In chapter 3 ("Why Not Just Test Everything?"), Weinberg has a section called "There are an infinite number of possible tests."

He talks about a backdoor placed into a highly secure program whereby the ordinary password protection could be bypassed by typing W followed by three spaces, then M followed by three spaces, then J followed by exactly 168 more keystrokes without once using the letter L.

Then he writes: "Do you get the point by now? If you didn't guess that the number of tests required to exhaustively test software is infinite, or at least "a number greater than I could run in my lifetime", you didn't understand the point of this chapter. Now you do."

Explain to your Project Manager that every day of additional testing will improve your confidence in your code, yet it can never reach 100%. Tell him that you would be happy to continue testing at the expense of other productive work. Then ask him how many more days he would like you to spend on testing.

If your Project Manager still doesn't get it, and you are feeling a bit flippant, ask him if he is 100% confident that every estimate he publishes is exactly correct and deadlines will never be missed. Ask him if he is 100% confident that no email he writes from now and forever will ever have a typo. Ask him if he is 100% confident that he will never make a mistake - now and in the future.

share|improve this answer
56  
+1: The project manager doesn't understand software well enough. Perhaps he needs to be educated. - Precisely. And you would know this as well as anyone, given your QA experience. –  Jim G. Mar 5 at 18:21
30  
You may wish to get a copy of Gerald Weinberg's excellent book "Perfect Software and other illusions about testing". In chapter 3 ("Why Not Just Test Everything?"), Weinberg has a section called "There are an infinite number of possible tests." - Yep. That book has aged very well. –  Jim G. Mar 5 at 18:21
2  
@Jim or Joe, if you have links to that book, feel free to edit it into the post to more permanently preserve it, as well as highlight it in the post body. Hope this helps. –  jmort253 Mar 6 at 6:37
1  
Its really just a time versus money issue. Even if it will require you to increase your confidence its still probably makes more sense for them to hire some QA. But, I guess that depends on how much the programmers are paid. I've generally considered it a cheaper proposition to hire QA than force developers to ensure high accuracy. Like that is even possible. –  EricLeaf Mar 6 at 16:46
2  
This is not actually a technical problem; it's a budgeting problem. This is a good thing, because he likely has more intuition for managing budgets than technical risk. So, honestly analyze how much time & money it would require to reach a 100% level of confidence, vs 99%, vs whatever reasonable level you are now targeting. Then show him the costs. He will decide 100% is not worth the spend. And yes, definitely refer to books or articles (e.g., on NASA's expensive development methods!) to show that you are illustrating a general and widely-understood fact of life, not just being negative. –  algal Mar 6 at 19:53

Boss: Are you 100% confident that this fix will not break any existing features or cause any adverse effects on user experience?

Me: No. Since we do not have a 100% coverage test suite there is no way to verify that any code change will avoid application breakage or adverse effects. However I've performed the following actions to assure that it is unlikely to perform in an unintended manner:

  • I've limited the scope of the change to just the module that it affects
  • I've read and updated the documentation accordingly
  • I've run the unit tests for this module and the affected modules
  • I've created unit tests to check new functionality added, and deprecated tests no longer relevant due to the change

While I cannot give you exactly 100% assurance, I've provided as much assurance as I can within the timeframe which I believe is reasonable for implementation of this functionality. In fact I've already reached the point of diminishing returns. I can spend another 5 hours to give you another 0.1% assurance, but it's already such a low chance that such effort isn't justified. However, you are in charge of the project and the timeframe, so let me know how much more time I should spend on this function.


Now the ball is in his court. If he really wants me to move my personal estimation from, say, 95% sure to 95.1% sure for a few more hours work, then hey - I can do that.

If he continues to be obnoxious about it, I'll have an email conversation with him and the "owner" of the project about this "100% tested" requirement, and what resources I believe are required to meet it.

share|improve this answer
4  
Deleted my answer when I saw that it was substantially the same as this one. –  outis nihil Mar 5 at 21:54
5  
I think this is a great answer. Except for the very last part. That works when you are a long time senior developer, but for junior or midlevel developer trying to bring the project owner into the fight could be devastating for a career. Getting on the wrong side of a PM with a good standing with management often results in being assigned to black hole projects and grunt work. Those are not career movers. Instead I would focus on explaining the costs of going to 98 99 and 100% –  ReallyTiredOfThisGame Mar 5 at 22:28
3  
@Chad I admit I've been blessed to only have had to deal with oppressive bureaucracy and middle management such as you describe for a short stint. In the remainder of my career, even as a junior, I've always been treated as a valued part of the team, and thus had no problems bringing up inconsistencies to those who were in a position to resolve them. I'm sure there are environments where this advice would prove less effective. –  Adam Davis Mar 5 at 22:39
1  
Great concept, but in execution you may want to shy away from statements like the following (emphasized): " I've provided as much assurance as I can within the timeframe which I believe is reasonable for implementation of this functionality" -- this being all the testing you can do given the schedule is (mostly) objective, stating what you find reasonable is likely just going to raise hackles. There are lots of similar sentences in the answer I would suggest you avoid. Explaining what you did, and what it would take to do more testing (along with the cost-benefit) is a great approach. –  jmac Mar 6 at 5:01
1  
+1: I've provided as much assurance as I can within the timeframe which I believe is reasonable for implementation of this functionality. In fact I've already reached the point of diminishing returns... so let me know how much more time I should spend on this function... Now the ball is in his court... Excellent. –  Vector Mar 9 at 8:47

As a developer, you are UNIT testing the changes to your code. In my opinions (as a developer), that is to check there are no obvious errors, everything compiles, all errors are trapped. You have no way of knowing what scenarios will be encountered once the code goes live (bad data, unexpected input, change in client software, the list is endless)

To have 100% confidence that a change will not affect anything, you would need a test environment that mirrors the live environment EXACTLY (data, hardware, permissions, connectivity) and a suite of test scripts that cover every single scenario. This, as is well known, is a virtually impossible environment to create for various reasons

If he is expecting 100% confidence, then he needs to provide the environment and manpower to back up his expectations

It is a little bit like when project managers and clients ask for stuff to be futureproof!

share|improve this answer
5  
Yea, a continuous integration environment is a must but if you have zero dedicated test engineers or QA personnel, asking for something to be 100% regression free is a little silly. –  Andrew Bartel Mar 5 at 16:24
2  
This is good information but I think the answer could more directly answer the question by explaining how the employee could bring this up or giving examples of what to say. –  starsplusplus Mar 5 at 16:44

It sounds like he's fallen into a bad habit. He's asking a question that he knows, at some level, is silly. But there's a bit of a compulsive element to it. My guess is that he's acting on an underlying anxiety, and asking an unreasonable question makes him feel more in control.

In these sorts of situations I try to diffuse the dynamic by being confident and inject a little humor. If you understand that his question isn't coming from reasoning, but from anxiety, then you can address that better indirectly than directly.

In situations like this, I usually allay the fears of management by presenting all the measures I've taken to ensure quality. He is, after all, the customer, and he needs to know that your priorities are in line with his. Look at these unit tests. Look at this monitoring. Look how the code is structured to keep changes local and modular. etc. If you convey a sense of confidence and control, it will allay his anxiety and you'll probably be able to have a more rational conversation.

That's where the art of this business comes into play. Not just "does it work", but does the customer feel good about it.

Ultimately, though, it's a business relationship. If the contracting arrangement is comfortable and profitable for you, then putting up with this quirk is worthwhile. It doesn't sound like he's all that serious about it, just kind of persistent. Like I say, he's developed an annoying habit. If he begins to react adversely and the tone becomes more hostile, then the balance of the business arrangement may make it not worthwhile for you. But from your short description, it sounds to me like you can still manage the relationship effectively.

share|improve this answer

Strictly speaking, one can never be 100% sure that a commit doesn't break a program.

Even with all sorts of testing possible (unit testing, integration, component, system, manual, UI, fuzz, security, penetration .. you name it). This is due to a Halting problem. A relevant extract from the Wikipedia follows below:

In computability theory, the halting problem can be stated as follows: "Given a description of an arbitrary computer program, decide whether the program finishes running or continues to run forever". This is equivalent to the problem of deciding, given a program and an input, whether the program will eventually halt when run with that input, or will run forever.

Alan Turing proved in 1936 that a general algorithm to solve the halting problem for all possible program-input pairs cannot exist. A key part of the proof was a mathematical definition of a computer and program, which became known as a Turing machine; the halting problem is undecidable over Turing machines.

If your PM cares about value and stable predictable delivery, you can perhaps convince him to have a look at SCRUM framework.

Others have given many interesting advices on how to interact with your PM. Personally, I would advise to set up a meeting with your PM and the team where you can discuss your processes. I am in strong favour of SCRUM, so this would be largely related to the SCRUM.

I would try explaining, that a goal of 100% is elusive. It cannot be reached. Never. In the whole universe. History has seen many such examples, demo of the release of Windows 95 for instance. Long time ago? Well, see how many red builds on a public CI server for open source projects; log in as guest if you don't have an account there. So an outcome of this - software will fail.

Second, I would advice to adopt a practice, where you can deliver value, instead of the confidence of a commit. Something, that customers care about. Iteratively, repeatedly and reliably. Now, you can shift your PM's perspective from the 100%-assurance, to something that actually matters. That is: software is in use, your product is improving and the team delivering value to the customer.

Third, the should be a definition of done. Something, that a development team comes up with. This may include: documentation, implementation, testing, quality gates etc. This is very important, as you can now shift the subjectivity (that is "are you 100% sure?") to objectivity (that is all bullet points of the definition of done are completed).

There are other very important steps that SCRUM promotes. But all of them would allow developers to mitigate such frustration, and allow them to deliver objectively quantifiable result, compared to subjective assurance.

share|improve this answer
3  
The halting problem doesn't allow deciding haltability of all programs, but it does allow creating programs which can be decided. And the idea of the PM is that he wants only such kind of programs. –  Paŭlo Ebermann Mar 7 at 1:14

The usual answer for this sort of goal is peer review and regression testing.

1) Don't commit anything to the production code stream until not just the author, but one or more other programmers, have sanity-checked it to make sure it changes only what is necessary, that it meets all the agreed-upon criteria for good coding practice practice, that it comes with a test that gives you decent odds of detecting whether a later change breaks the logic again, and so on.

2) Don't commit anything to the production code stream until ALL the regression tests have been rerun and it has been proven not to break anything which the test for. If any failure occurs during that test, the change must be backed out until it can be clearly established not to be have caused the problem.

3) Alpha- and beta-test early and often with real-world customer scenarios. Customers will do things with your code that you never expected.

None of those is 100%, nor is their combination. But they get you considerably closer. They do require a nontrivial investment of additional resources. They do slow down development compared to skunkworks "just do it and we'll fix it when it breaks" coding practice. But if you really want bug-free code, recognizing that humans are fallable and putting practices in place to help catch failures before they reach the customers may be more than worth those costs.

I'm told that there was never a bug reported in the code IBM wrote for NASA -- because it was peer-reviewed and tested to death during design, during development, and before being released.

If you're doing something life-critical, it's obviously worth that investment. If you're doing something which is infrastructure for large businesses, it's worth that investment; you don't want to be the one whose bug takes down a billion-dollar company's business processes.

Yes, it drives good coders crazy. Until the first time it saves their behinds.

share|improve this answer

A lot of people have described this as an education problem, and I'm not saying they're wrong.

It's also possible that it's a political problem. What the question could actually mean, is that the project manager wants to be absolved of responsibility for any errors. He gets a sworn statement from you that he feels he can "reasonably" rely on, so if anything goes wrong he says that he's done everything he could to prevent it but you failed.

This is not good management and it's also not 100% guaranteed to put him in the clear if trouble does happen.

I admit this is speculation on my part, but butt-covering exercises are not at all uncommon in professional life and you have to deal with them. So if this rings true, then what you need to do to solve the problem is not just to educate the manager that 100% confidence is impossible. You also need to convince him that a bug is not a disaster -- for him personally or for the company.

This might mean for example providing examples of bugs being dealt with at reasonable cost and without any blame flying around. It might mean looking at his own company culture, whether there's someone else in the company gearing up to lay unjust blame on him if anything goes wrong. It might also mean putting in procedures to deal with future bugs as quickly and inexpensively as possible. If the company really was 100% confident in the code then such procedures would be pointless because it would be prepared to bet arbitrarily large amounts of money on there being no future bugs!

As the ultimate sanction, if he (the employer) is asking you (the contractor) to sell him an assurance that you cannot give about your work, and nothing will change his mind on this point, then all you can do is clearly state that you are not able to provide that assurance, and that he's welcome to employ someone else if he thinks there's someone who can. Of course that's a long way down the road, but you might as well know your BATNA before you start.

share|improve this answer

If he is in fact micro-managing you and looking over your shoulder throughout the process of building the project, there's an easy way to make sure that you can 'guarantee' 100% confidence without having him hawk you about it later.

Get him to approve it himself

To be clear, you shouldn't make this as a demand, but rather a suggestion, that if he truly wants 100% perfect code, then he should look at what you are doing himself and approve every change you are making along the way. This isn't to say he should literally read the code, but rather to see it in action and confirm 'yes, that is how it should act'.

If you are the sole tester of your own code, this is not unreasonable - you're already fully focused on the project, and if he wants perfection, he should be taking responsibility for ensuring that perfection himself.

Also, take notes on everything he approves, so that later when it does inevitably break, you can point to your documentation showing that he approved it.

If, for any reason, you don't think this would go over well (for instance, if asking him to do more work is something you already know would be disasterous), all you can really do is as much hard error testing as you can, write in your reports everything that you know works correctly, and give him 100% assurance that, 'within the limits of my testing, I am 100% confident in these changes'.

Unfortunately, you may be in the position of having a 'boss' whose understanding of your work is limited, which is always a pain when trying to explain how error correction is impossible to maintain with 100% confidence. So to summarize, your best bet might just be to do the best you can, record everything, and make it understood that you're doing what you can within your own limits.

share|improve this answer
2  
Do you really think this is a good way to build a strong relationship with your PM? It seems to me that this type of response is rather passive aggressive and could result in blow-back. –  ReallyTiredOfThisGame Mar 5 at 20:05

Quite some nice answers here, the PM definetly needs to be educated and learn a bit about what it means to write software.

I do not want to repeat anything of that, but throw in another, rather unusual idea. This method is rather risky and depends very much on how high your reputation is, how good your skills are (or more how they are perceived), and both your personality and that of your PM. I am presuming that you did not misunderstand him, and he really means 100% (I often see people saying 100%, but really meaning "do the best you can")

It worked for me once, and that is the only reaons I am mentioning it here. You have been warned. This is mostly a possible way to educate a PM if all other means fail.

Sometimes, a PM (or any other manager) just doesn't want to listen, so you must somewhere make him (or the team) hit a wall to stop for a moment and think. In this scenario it means: Work as good as you can, try to test as good as you can. Give your very best, and then say "Yes, I am 100% sure that this will work".

If you are extremely lucky, no bug will ever occur and everyone is happy; but in reality what will happen is: there is a bug. What shall you do now? You admit that you made a mistake. Make a connection with bugs and the mistake of being 100% sure. Humans are imperfect and can create bugs in software. Humans are imperfect and create bugs in tests. Consequently, humans are imperfect and can "create bugs" in their perception of being 100% sure that there is no bug.

Present this well, and 100% watertight (haha, j/k, the 100% again). Make sure that after all of this, the message that went over is "I can not be 100% confident in my tests". If the PM then can't make the logical step of "This will be the same for all developers" then all hope is probably lost though...

But please, do try other things first!

share|improve this answer

The key indicator is that this is a recent change. Something (or someone) has probably given your PM a bad experience, and now he's on edge every time something changes. Or maybe he read something in a book or magazine.

If you can get the PM to tell you his story (perhaps over his beverage of choice) then you can sympathize, and he may become receptive to "innovation" a.k.a sound software-engineering practice.

This is your chance to talk about human error and the ways this industry has found to increase the levels of confidence in our designs and code. To talk about the trade-offs in confidence, quality, resources, and schedule that come from different approaches to testing, peer code review, formal methods (a.k.a. correctness-by-construction).

Speak his language: Use metaphor to illustrate the size of the problem. Is it 40,000 lines of code? Tell him it's like a 600 page murder mystery in a foreign language. If you want to change something in the middle of chapter 12, how do you make sure it doesn't cause a break in continuity with the rest of the story?

Look for buy-in on ways to increase confidence towards an acceptable target within reasonable economic boundaries. You won't get to implement SEI Capability Maturity Model level 5 overnight. But you might be able to go from the current question to "does the automated regression test suite pass?" and "how do we express this new requirement in the regression test system?"

share|improve this answer

I'd answer it in the most mathematical way, considering that he is asking for a probability with 100% confidence, and completely ignoring the effect that statistical distributions would have on such number: You should give him 2 or 3 numbers, with associated confidence: 1 week at 90%, 2 weeks at 95%, 6 months at 100%. The last number was an exaggeration but I'm sure you got the point.

See Wikipedia's article on Confidence Intervals for more reading.

share|improve this answer
2  
cc @gnat - Marcus, a good suggestion in this case would be to add to your answer some details on what type of tone to use. I can see how this could come off as sarcastic, and that would likely result in even less cooperation with the PM. People don't generally work with you when you've made them feel stupid. :) Good luck and hope this helps! –  jmort253 Mar 6 at 7:06

protected by enderland Mar 6 at 17:07

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.