Project Manager asks for complete 100% confidence everytime committing code

Question

I've got an ongoing relationship with a long term business partner as a consultant where his role is project manager (task manager + direction), and my role is a contracted developer. He has a tendency to micromanage my time with his tasks and oversight, but also has a strong sense of perfection.

Recently with every single programming task undertaken he is asking me to confirm that I have "100% confidence that this fix will not break any existing features or cause any adverse effects on user experience". If I can't affirm that, he assumes I haven't tested it well enough or should go check it again. And yes, he actually asks this every single bug fix, it isn't just implied.

As a developer, I do test my work on multiple unit cases, but can't say that it is possible to fully regression test the entire product for each 2-hour task I accomplish. There is no QA team either. The product has lots of intermingled parts throughout (not just self-contained pages), some 40,000 lines of code written over 4 years, and sometimes unexpected things happen that we weren't even aware of. I sense he sees this as poor testing.

How should I respond to his question in this case, without seeming incompetent? Honestly I never have 100% confidence sitewide, but I do have confidence in my testing methods. And, as a developer I also know that it isn't uncommon for unexpected bugs to emerge later from these core changes.

EDIT:
I'm not necessarily looking for a solution to make this 100%, as our group doesn't have the time or resources to implement a full QA process or get into setting up automated solutions. I'm looking for how to interact with the manager around existing work, especially when he isn't entirely a technical person himself. He isn't a programmer.

Answer 1

How should I respond to his question in this case, without seeming incompetent? Honestly I never have 100% confidence sitewide, but I do have confidence in my testing methods. And, as a developer I also know that it isn't uncommon for unexpected bugs to emerge later from these core changes.

The project manager doesn't understand software well enough, and certainly doesn't understand testing well enough. Perhaps he needs to be educated.

If you had a professional QA Department, I'd tell you to enlist the support of the QA Manager in explaining to this Project Manager the nature of software, the nature of bugs, and the nature of testing. I'd have the QA Manager indicate why it is simply not possible to test every condition, and how releasing/not releasing is a business activity aided by the findings from testing, but never by perfect information.

You may wish to get a copy of Gerald Weinberg's excellent book "Perfect Software and other illusions about testing". In chapter 3 ("Why Not Just Test Everything?"), Weinberg has a section called "There are an infinite number of possible tests."

He talks about a backdoor placed into a highly secure program whereby the ordinary password protection could be bypassed by typing W followed by three spaces, then M followed by three spaces, then J followed by exactly 168 more keystrokes without once using the letter L.

Then he writes: "Do you get the point by now? If you didn't guess that the number of tests required to exhaustively test software is infinite, or at least "a number greater than I could run in my lifetime", you didn't understand the point of this chapter. Now you do."

Explain to your Project Manager that every day of additional testing will improve your confidence in your code somewhat, yet it can never reach 100%. Tell him that you would be happy to continue testing at the expense of your other productive work. Then ask him how many more days he would like you to spend on testing and which of your other work should be deferred.

If your Project Manager still doesn't get it, and you are feeling a bit flippant, ask him if he is 100% confident that every estimate he publishes is exactly correct and deadlines will never be missed. Ask him if he is 100% confident that no email he writes from now through forever will ever have a typo. Ask him if he is 100% confident that he will never make a mistake - now and in the future.

Answer 2

Boss: Are you 100% confident that this fix will not break any existing features or cause any adverse effects on user experience?

Me: No. Since we do not have a 100% coverage test suite there is no way to verify that any code change will avoid application breakage or adverse effects. However I've performed the following actions to assure that it is unlikely to perform in an unintended manner:

• I've limited the scope of the change to just the module that it affects
• I've read and updated the documentation accordingly
• I've run the unit tests for this module and the affected modules
• I've created unit tests to check new functionality added, and deprecated tests no longer relevant due to the change

While I cannot give you exactly 100% assurance, I've provided as much assurance as I can within the timeframe which I believe is reasonable for implementation of this functionality. In fact I've already reached the point of diminishing returns. I can spend another 5 hours to give you another 0.1% assurance, but it's already such a low chance that such effort isn't justified. However, you are in charge of the project and the timeframe, so let me know how much more time I should spend on this function.

---

Now the ball is in his court. If he really wants me to move my personal estimation from, say, 95% sure to 95.1% sure for a few more hours work, then hey - I can do that.

If he continues to be obnoxious about it, I'll have an email conversation with him and the "owner" of the project about this "100% tested" requirement, and what resources I believe are required to meet it.

Answer 3

As a developer, you are UNIT testing the changes to your code. In my opinions (as a developer), that is to check there are no obvious errors, everything compiles, all errors are trapped. You have no way of knowing what scenarios will be encountered once the code goes live (bad data, unexpected input, change in client software, the list is endless)

To have 100% confidence that a change will not affect anything, you would need a test environment that mirrors the live environment EXACTLY (data, hardware, permissions, connectivity) and a suite of test scripts that cover every single scenario. This, as is well known, is a virtually impossible environment to create for various reasons

If he is expecting 100% confidence, then he needs to provide the environment and manpower to back up his expectations

It is a little bit like when project managers and clients ask for stuff to be futureproof!

Answer 4

It sounds like he's fallen into a bad habit. He's asking a question that he knows, at some level, is silly. But there's a bit of a compulsive element to it. My guess is that he's acting on an underlying anxiety, and asking an unreasonable question makes him feel more in control.

In these sorts of situations I try to diffuse the dynamic by being confident and inject a little humor. If you understand that his question isn't coming from reasoning, but from anxiety, then you can address that better indirectly than directly.

In situations like this, I usually allay the fears of management by presenting all the measures I've taken to ensure quality. He is, after all, the customer, and he needs to know that your priorities are in line with his. Look at these unit tests. Look at this monitoring. Look how the code is structured to keep changes local and modular. etc. If you convey a sense of confidence and control, it will allay his anxiety and you'll probably be able to have a more rational conversation.

That's where the art of this business comes into play. Not just "does it work", but does the customer feel good about it.

Ultimately, though, it's a business relationship. If the contracting arrangement is comfortable and profitable for you, then putting up with this quirk is worthwhile. It doesn't sound like he's all that serious about it, just kind of persistent. Like I say, he's developed an annoying habit. If he begins to react adversely and the tone becomes more hostile, then the balance of the business arrangement may make it not worthwhile for you. But from your short description, it sounds to me like you can still manage the relationship effectively.

Answer 5

A lot of people have described this as an education problem, and I'm not saying they're wrong.

It's also possible that it's a political problem. What the question could actually mean, is that the project manager wants to be absolved of responsibility for any errors. He gets a sworn statement from you that he feels he can "reasonably" rely on, so if anything goes wrong he says that he's done everything he could to prevent it but you failed.

This is not good management and it's also not 100% guaranteed to put him in the clear if trouble does happen.

I admit this is speculation on my part, but butt-covering exercises are not at all uncommon in professional life and you have to deal with them. So if this rings true, then what you need to do to solve the problem is not just to educate the manager that 100% confidence is impossible. You also need to convince him that a bug is not a disaster -- for him personally or for the company.

This might mean for example providing examples of bugs being dealt with at reasonable cost and without any blame flying around. It might mean looking at his own company culture, whether there's someone else in the company gearing up to lay unjust blame on him if anything goes wrong. It might also mean putting in procedures to deal with future bugs as quickly and inexpensively as possible. If the company really was 100% confident in the code then such procedures would be pointless because it would be prepared to bet arbitrarily large amounts of money on there being no future bugs!

As the ultimate sanction, if he (the employer) is asking you (the contractor) to sell him an assurance that you cannot give about your work, and nothing will change his mind on this point, then all you can do is clearly state that you are not able to provide that assurance, and that he's welcome to employ someone else if he thinks there's someone who can. Of course that's a long way down the road, but you might as well know your BATNA before you start.

Answer 6

Strictly speaking, one can never be 100% sure that a commit doesn't break a program.

Even with all sorts of testing possible (unit testing, integration, component, system, manual, UI, fuzz, security, penetration .. you name it). This is due to a Halting problem. A relevant extract from the Wikipedia follows below:

In computability theory, the halting problem can be stated as follows: "Given a description of an arbitrary computer program, decide whether the program finishes running or continues to run forever". This is equivalent to the problem of deciding, given a program and an input, whether the program will eventually halt when run with that input, or will run forever.

Alan Turing proved in 1936 that a general algorithm to solve the halting problem for all possible program-input pairs cannot exist. A key part of the proof was a mathematical definition of a computer and program, which became known as a Turing machine; the halting problem is undecidable over Turing machines.

If your PM cares about value and stable predictable delivery, you can perhaps convince him to have a look at SCRUM framework.

Others have given many interesting advices on how to interact with your PM. Personally, I would advise to set up a meeting with your PM and the team where you can discuss your processes. I am in strong favour of SCRUM, so this would be largely related to the SCRUM.

I would try explaining, that a goal of 100% is elusive. It cannot be reached. Never. In the whole universe. History has seen many such examples, demo of the release of Windows 95 for instance. Long time ago? Well, see how many red builds on a public CI server for open source projects; log in as guest if you don't have an account there. So an outcome of this - software will fail.

Second, I would advice to adopt a practice, where you can deliver value, instead of the confidence of a commit. Something, that customers care about. Iteratively, repeatedly and reliably. Now, you can shift your PM's perspective from the 100%-assurance, to something that actually matters. That is: software is in use, your product is improving and the team delivering value to the customer.

Third, the should be a definition of done. Something, that a development team comes up with. This may include: documentation, implementation, testing, quality gates etc. This is very important, as you can now shift the subjectivity (that is "are you 100% sure?") to objectivity (that is all bullet points of the definition of done are completed).

There are other very important steps that SCRUM promotes. But all of them would allow developers to mitigate such frustration, and allow them to deliver objectively quantifiable result, compared to subjective assurance.

Answer 7

The usual answer for this sort of goal is peer review and regression testing.

1) Don't commit anything to the production code stream until not just the author, but one or more other programmers, have sanity-checked it to make sure it changes only what is necessary, that it meets all the agreed-upon criteria for good coding practice practice, that it comes with a test that gives you decent odds of detecting whether a later change breaks the logic again, and so on.

2) Don't commit anything to the production code stream until ALL the regression tests have been rerun and it has been proven not to break anything which the test for. If any failure occurs during that test, the change must be backed out until it can be clearly established not to be have caused the problem.

3) Alpha- and beta-test early and often with real-world customer scenarios. Customers will do things with your code that you never expected.

None of those is 100%, nor is their combination. But they get you considerably closer. They do require a nontrivial investment of additional resources. They do slow down development compared to skunkworks "just do it and we'll fix it when it breaks" coding practice. But if you really want bug-free code, recognizing that humans are fallable and putting practices in place to help catch failures before they reach the customers may be more than worth those costs.

I'm told that there was never a bug reported in the code IBM wrote for NASA -- because it was peer-reviewed and tested to death during design, during development, and before being released.

If you're doing something life-critical, it's obviously worth that investment. If you're doing something which is infrastructure for large businesses, it's worth that investment; you don't want to be the one whose bug takes down a billion-dollar company's business processes.

Yes, it drives good coders crazy. Until the first time it saves their behinds.

Answer 8

The truth is that it is poor testing. The reality is that a company unwilling to invest in a QA team is going to have poor testing. Good testing is expensive and takes time. The company has accepted the risk by not authorizing the time and money.

Even a QA team can't guarantee every possibility is tested because the possible paths through a complex program are basically infinite. However, they will get you far closer than you are right now. One reason why is that it is impossible for a dev to adequately test their own code. They know what it does, so they tend to design tests around what they think it is supposed to do. They miss edge cases, they miss stupid things that users do that a dev would never do because they know how it works, they sometimes interpret the requirement incorrectly but all of their tests will reflect their original incorrect interpretation. They often miss errors in the requirement and do what they are asked to do not what they should have done (this is the cause of a huge number of bugs that are found only after the actual users [who are all too often not consulted in defining the requirement] try to use the software). They miss effects on parts of the application they have never had reason to work in especially parts that are done by specialists (such as a table change that makes sense for the application but breaks an automated import process or a report.)

If he wants higher quality he will have to pay for it in both time and money. Even with full QA you cant get to 100% although certainly NASA and its contractors come close. They also spend a great deal more money than your company is spending. Even then, they managed to completely miss MARS one time.

If what he wants is assurance that any issues won't cause him harm with the clients, then talk about your process for testing (Show him the list of tests you did run.), what you think would be affected and how you checked that, your process for how you would revert a bad push and your process for logging errors so that you will see them before most clients notice them. Give him confidence that even if there is a problem, it can be fixed. Talk about the value of getting the code (new feature or fix) out quickly vice the additional time it would take to test more thoroughly. Talk about the risks of not getting it out quickly.

You can also ask him to provide the thorough regression test of the system every time you make a change because it is not possible for a dev to completely test his own work (you know what your assumptions were, if they are not right, you would never test for that.) Make sure he knows he will have to test every single page of the application and and every single thing that could be done on a page in every possible order. Oh yeah, test any imports/exports, reports, automated jobs as well. And any related applications that might be affected. Once he has tried to thoroughly exercise the system once, he will realize why you can't make that assurance.

Another thing to try is to tell him up front that no you can't make that guarantee, but if he authorizes X testing hours, you can get closer to making that guarantee. Give him an itemized list of the the additional tests and the hours it would take to design and run each one. Tell him what percentage of confidence you would have after executing all those tests and what percentage of confidence you have right now.

For that matter tell him how long it would take just to run all the current unit tests you have regardless of whether they are related to this issue or not. So if you currently have 1000 unit tests and each one takes five minutes to set up and run and evaluate the results that would be 83 hours. Ask him for the authorization to go ahead and expend that time.

Answer 9

The product has lots of intermingled parts throughout (not just self-contained pages), some 40,000 lines of code written over 4 years, and sometimes unexpected things happen that we weren't even aware of. I sense he sees this as poor testing.

This is your problem, but up until now is not your fault. If you don't fix it, you'll have to take responsibility at some point. Have a discussion with your boss that until this is fixed, you will never be 100%. Suggest refactoring. Also, 100% in the mind of non-technical people isn't the same as a programmer's. Maybe you should indicate you're "100% sure the client probably won't notice."

There's no QA Team This is a luxury your company can't afford, so you're it. Imagine there was a QA Team (you) and determine how long they would take to test your code, then put that in your estimation. There's no way you can code and QA simultaneously, so you can't do in parallel.

Stop being so anxious to check in code and meet what appears to be unreasonable demands. Give them what the want. Once they find out the cost (excessive coding time), he may change his mind.

Answer 10

If he is in fact micro-managing you and looking over your shoulder throughout the process of building the project, there's an easy way to make sure that you can 'guarantee' 100% confidence without having him hawk you about it later.

Get him to approve it himself

To be clear, you shouldn't make this as a demand, but rather a suggestion, that if he truly wants 100% perfect code, then he should look at what you are doing himself and approve every change you are making along the way. This isn't to say he should literally read the code, but rather to see it in action and confirm 'yes, that is how it should act'.

If you are the sole tester of your own code, this is not unreasonable - you're already fully focused on the project, and if he wants perfection, he should be taking responsibility for ensuring that perfection himself.

Also, take notes on everything he approves, so that later when it does inevitably break, you can point to your documentation showing that he approved it.

If, for any reason, you don't think this would go over well (for instance, if asking him to do more work is something you already know would be disasterous), all you can really do is as much hard error testing as you can, write in your reports everything that you know works correctly, and give him 100% assurance that, 'within the limits of my testing, I am 100% confident in these changes'.

Unfortunately, you may be in the position of having a 'boss' whose understanding of your work is limited, which is always a pain when trying to explain how error correction is impossible to maintain with 100% confidence. So to summarize, your best bet might just be to do the best you can, record everything, and make it understood that you're doing what you can within your own limits.

Answer 11

Quite some nice answers here, the PM definetly needs to be educated and learn a bit about what it means to write software.

I do not want to repeat anything of that, but throw in another, rather unusual idea. This method is rather risky and depends very much on how high your reputation is, how good your skills are (or more how they are perceived), and both your personality and that of your PM. I am presuming that you did not misunderstand him, and he really means 100% (I often see people saying 100%, but really meaning "do the best you can")

It worked for me once, and that is the only reaons I am mentioning it here. You have been warned. This is mostly a possible way to educate a PM if all other means fail.

Sometimes, a PM (or any other manager) just doesn't want to listen, so you must somewhere make him (or the team) hit a wall to stop for a moment and think. In this scenario it means: Work as good as you can, try to test as good as you can. Give your very best, and then say "Yes, I am 100% sure that this will work".

If you are extremely lucky, no bug will ever occur and everyone is happy; but in reality what will happen is: there is a bug. What shall you do now? You admit that you made a mistake. Make a connection with bugs and the mistake of being 100% sure. Humans are imperfect and can create bugs in software. Humans are imperfect and create bugs in tests. Consequently, humans are imperfect and can "create bugs" in their perception of being 100% sure that there is no bug.

Present this well, and 100% watertight (haha, j/k, the 100% again). Make sure that after all of this, the message that went over is "I can not be 100% confident in my tests". If the PM then can't make the logical step of "This will be the same for all developers" then all hope is probably lost though...

But please, do try other things first!

Answer 12

The key indicator is that this is a recent change. Something (or someone) has probably given your PM a bad experience, and now he's on edge every time something changes. Or maybe he read something in a book or magazine.

If you can get the PM to tell you his story (perhaps over his beverage of choice) then you can sympathize, and he may become receptive to "innovation" a.k.a sound software-engineering practice.

This is your chance to talk about human error and the ways this industry has found to increase the levels of confidence in our designs and code. To talk about the trade-offs in confidence, quality, resources, and schedule that come from different approaches to testing, peer code review, formal methods (a.k.a. correctness-by-construction).

Speak his language: Use metaphor to illustrate the size of the problem. Is it 40,000 lines of code? Tell him it's like a 600 page murder mystery in a foreign language. If you want to change something in the middle of chapter 12, how do you make sure it doesn't cause a break in continuity with the rest of the story?

Look for buy-in on ways to increase confidence towards an acceptable target within reasonable economic boundaries. You won't get to implement SEI Capability Maturity Model level 5 overnight. But you might be able to go from the current question to "does the automated regression test suite pass?" and "how do we express this new requirement in the regression test system?"

Answer 13

I'd answer it in the most mathematical way, considering that he is asking for a probability with 100% confidence, and completely ignoring the effect that statistical distributions would have on such number: You should give him 2 or 3 numbers, with associated confidence: 1 week at 90%, 2 weeks at 95%, 6 months at 100%. The last number was an exaggeration but I'm sure you got the point.

See Wikipedia's article on Confidence Intervals for more reading.