How to diplomatically resolve an unreproducible bug your manager wants addressed?

Question

I work in the software industry in a junior capacity and have been unable to reproduce a bug.

This issue has apparently been seen < 3 times at a customer site, and a senior member of the team have heard of it, however they have not been able reproduce or isolate the cause. Unfortunately I have not been provided any logs.

We normally close off such a bug with the stated reason "unable to reproduce". I have spent more than 2 days digging through code, comparing versions, attempting to reproduce this bug by following the given steps to reproduce. Unfortunately with no more additional useful information at hand, I don't see or know of anything else useful I can really do.

My manager has suggested I try different things than what was given in the steps to reproduce, to try and reproduce the bugs behavior I have to, no avail. My manager at the least wanted an answer to what possibly cause it. (This makes me somewhat uncomfortable as what "might cause it" when that can be definitively tested quickly, it either will or won't)

I would like to provide a satisfying answer to my manager, and frankly I don't have one. My manager believes that if it's happened twice on a customer system (but no useful logs pulled or available to me) that it is bound to happen again (I can agree with that however without additional information I don't know what to try), however I have been unable to reproduce this bug.

My work on the bug has the same conclusion as the senior member who has heard of this behavior (couldn't reproduce or isolate the cause).

I don't want to spend anymore time spinning my wheels on this as I feel I've given it every reasonable effort with the information I've been given. Is there a good way to sell this to my manager?

Clarifications:

The system at hand is a deployment of our companies software solution for a customer. I don't wish to go into anymore detail into this, my apologies.

By no logs I mean that no logs were provided from when these scenario(s) happened. Our code logs errors, warnings, etc. We have in my opinion decent logging.

In my opinion the bug's severity should be low to medium. The client has to wait longer to see the resolution of an action, however I still defer to those around me who know better than me, still want to see this addressed.

Answer 1

You have three entirely separate problems: the technical thing of finding the bug, the political thing of keeping your manager happy, and the different political thing of keeping the client happy. (For example, your boss may want your other work to still get done, but the client doesn't care about that. They care about other stuff.)

You want your boss and the client to know that you care and want to fix the bug. But you don't want to give the impression that you're lost and don't know how to find it. The longer you work on it, the higher the chance people will think you're not good at this. But refusing to work on it is bad too: you have to care and to want to fix it.

So. They have told you that something bad happens. An error message appears, or a wrong value appears somewhere. And you've looked for that error message or for bits of the code that affect that value, but you can't tell which of those ran. When you do what they say to do, it doesn't happen, but you can't see a significant reason why it "works on my machine." Make sure that your boss understands both of these things. That you are thinking logically and you have looked into the possible causes from looking through the code, and that you have tried to repro with their instructions.

Next (check this with your boss if need be) go to the client and get someone to agree to be your go-between on this bug. The person has to agree to put in some time for you so that you can solve it. Follow steps like these:

1. the person or you gets someone in the client org to agree to give you logs on request.
2. the person triggers the bug. They take screenshots for you. If there's an exception dialog, they click for More Details and they copy and paste everything they get. If there's a report or other save-able exportable asset that demonstrates the problem (eg the bad number) they save or export it and give it to you.
3. the person or you asks for logs covering the time that just happened.
4. you investigate armed with this info.

If you solve the bug, yay. If not, you create an instrumented build. Say a particular error message occurs in several spots in your code. You change it so each of those spots leaves a unique signature - in the logs, in the message, somehow. Maybe you also turn the logging up if that's a thing you can do. You get that code to your person. Then steps 2 through 4 again.

Still not found? You check that your person is ok with still being your person. You check with your boss that it's ok to still be looking into it. You add more instrumentation, more logging, more capturing. Your boss sees that you're not just thrashing, but you're logically working your way towards narrowing it down. Your client, who is interacting with you a lot, sees too that you're working on this and working intelligently. And you are likely to solve it this way, too. But along the way you will be sending the right message to those who are watching you.

Still not found? There are things you can install on the client's machine that enable "historical debugging" or "replay debugging" -- if the client refuses to install it you can honestly say "well I won't be able to solve it if you don't co-operate" and if they do install it, well, again you're sending the right message and you are that much more likely to solve it.

The key is to know what you need (more information) and how to get it. Then set to work getting it and make sure everyone knows you have a plan and are following it.

Answer 2

The answer depends from the severity of the problem :

High Severity

If the problems is severe and creates real problems with high impact on customers then you have to continue working on the problem until you find a solution or a reasonable workaround that will minimize the consequences of the bug. You may try the following:

• Add additional logging
• Work on the machine that produced the problem
• Talk with the person who produced the problem
• Make your code more robust

Low severity

If the consequences of the bug are minor, then you have to convince your boss that the cost of fixing the problem is much greater than the benefits. Then the manager will have to handle the customer.

If you cannot convince your manager then say to him, that you will work on the problem for one hour every day, and that you cannot estimate when this problem will be solved. This will give you three benefits :

• You will work on your other tasks
• Not working continuously on the problem may help you to find the bug more easily
• After one month your boss will see things differently and probably will be easier to convince him to stop working on the bug

Answer 3

If you can't reproduce the bug, how will you know if you fixed it? All bugs should have a test script (set of instructions) that will reproduce the bug. The test script is run before and after your bug fix is installed. This will tell that you fixed the problem. Enforcing this is as simple as giving bugs that don't have a test script a low priority.

Answer 4

Talk to your manager. Tell him that this bug is hard to reproduce (and you haven't managed at all so far), and that you have no useful information to help you. And that you have several choices: Go on trying to reproduce the bug, examine the code that is involved and try finding bugs there, or modifying the code to make it easier to find the bug, through logging and whatever suitable measures. Or of course do nothing - mark the bug as "cannot reproduce" and hope it doesn't come back.

Your manager should then decide which path he wants you to go. Fattie's answer might help fixing the problem, but might get you into trouble if your manager has totally different priorities.

PS. A colleague of mine had a bug that would only strike after 18:12 in the evening (just for fun, try to figure out what's special about 18:12:16 in the evening). Everything worked absolutely fine before that. From nine to five, that bug was absolutely not reproducible.

PS. There is one reasonably common bug that is repeated by developers again and again and results in software malfunctioning in up to the first three days of the year, displaying the wrong year. But not every year. On January 4th that bug disappears.

Answer 5

For arguments sake lets assume somebody's cat died as a result of this bug. This person wants you to find what is responsible to ensure no more cats must perish because of this senseless maniac bug, who is practically a psychopathic serial cat killer, with no conscious.

Now if this was your cat, at what point would you want the detective to stop looking? What if they had exhausted all reasonable leads after 1 day, I think you would still want them to keep looking so you still have a chance of getting justice for your cat.

But the fact is you have no emotional attachment to this cat, and you know deep down your efforts are in vein. So what to do? IMHO an appropriate course of action would be to reassure all parties that you will do what you can to catch this killer, but this may need to enter a long term strategy also to take any preventive measures as a result of your investigation, to ensure cats are as safe as they can be in case the killer were to strike again. Then you will need to get additional work assigned to you whilst you still search for the cat killer. At the beginning this might not be much work and your time will be still mainly on chasing the killer. But slowly you can ramp up your additional tasks till you hardly spend any direct time on the case.

Do keep it in the back of your mind, maybe suggest additional logging changes you could make to help set a trap for this bug if it does come up in the future. That will probably be your best bet in ever really solving the bug as you say currently there is no logs, maybe logging is something lacking in the business which could be improved and you could spend time on that and it would still be under the hood of catching the bug, and then move on to the next task once you convince your boss that if the cat killer strikes again you will catch him then, and minimize any harm done to customers in this process.

Answer 6

You said that your software produces adequate logs. Why don't you have these logs?

If it's because you don't have the security permissions to see them or something administrative like that, tell your boss that you need to see the logs to be able to solve the problem. If he can't give you access but he can give somebody else access, then maybe that person should be assigned to work on the problem.

If you don't have the logs because they were deleted between when the bug happened and when you were told about the problem, then you may be limited to saying, Wait until it happens again and this time capture the logs before they get deleted.

If the logs aren't adequate to diagnose the problem, the only solution may be to improve the logging.

You can ask the user to be on the lookout for this to happen again and if and when it does to get screen shots or whatever information you think may be useful.

You can always try poring over the code some more and doing more testing on your own, but you indicate that you've done this and it hasn't solved the problem. The only suggestion I can give on that point is: Programs are often bad at testing their own code or the code of other programmers because they go through the steps as intended, while a real user who doesn't understand the design philosophy may do crazy stuff. Just for example, I recently had a problem where for some reason a user decided to type his entire address -- street, city, state, zip -- into the zip code field. We never considered someone doing this when writing our zip code validation function and we got bizarre results. Not saying that particular craziness, of course, but think outside the box. I have no idea what your app does, so I can't give any specific suggestions.