28

I'm a front end dev for a company that makes websites for clients.

I have a pretty thorough testing process that is already quite time consuming, but I always test fully before anything goes live.

On a very rare occasion something small will slip through and the project manager will get mad and declare that I should have caught it in testing. Its usually nothing fundamental to the function of the site but a style issue on a page I did not test.

I feel that if I tested every page of every website on every browser the time involved to do this would make it impossible to complete all of my work.

Is it reasonable that my project manager expects no bugs to make it to a live site?

  • If bug-free software is reasonable, what process is used to produce it? And how would I explain the costs of this process to a manager and balance the amount of testing expected with a large workload?
  • If bug-free software is not reasonable, how can I convince a manager of this?

closed as off-topic by gnat, Lilienthal, Michael Grubey, Masked Man, Thalantas Feb 8 '17 at 9:28

This question appears to be off-topic. The users who voted to close gave this specific reason:

If this question can be reworded to fit the rules in the help center, please edit the question.

  • 17
    See: Is it possible to reach absolute zero bug state for large scale software? on softwareengineering.SE. In short: no. – sleske Feb 7 '17 at 18:17
  • 22
    Even for small projects, being bug-free is impossible. Not only is there always something (no matter how small) that gets overlooked, in practice requirements are never so clear-cut that one cannot find something to call a "bug" that is simply not covered by requirements. – sleske Feb 7 '17 at 18:25
  • 19
    Seems to me like on the tech side you have a flawed testing methodology ("I didn't get around to testing page X" is .. silly. It sounds both manual, as well as incomplete. Automation is the key here). On the management side it seems like people have both completely unreasonable expectations, as well as a fundamental lack of understanding of testing and software development. That's why companies which know what they're doing have something called "user acceptance testing". It means that the "target audience" (aka customer) uses the damned thing and points out anything that doesn't meet specs – AndreiROM Feb 7 '17 at 19:02
  • Comments are not for extended discussion; this conversation has been moved to chat. – Jane S Feb 8 '17 at 13:21
  • 2
    You know who aims for 0 bugs in production? NASA... I'm guessing your boss isn't willing to support the additional development cost and testing cost in order to mimic NASA's software development model. So, no, it's not reasonable to expect 0 bugs in production. EDIT: And even NASA still gets bugs in production! – Maybe_Factor Feb 9 '17 at 0:42
48

Is it reasonable that my project manager expects no bugs to make it to a live site,

No.

or is it reasonable to say I cant always check everything or guaruntee there will never be a bug?

Yes, it is. Otherwise you would never ship anything.

And how would I balance the amount of testing expected with a large workload?

Now we are getting somewhere :-). That is an excellent question to ask. This is what you should ask your project manager: it is their job to decide whether more features or higher quality are more important.

Hopefully your manager understands that there is a tradeoff involved - it's the old "Project triangle" (cost - scope - time, pick any two), just with "quality" as an added dimension.

If your PM does understand there is a tradeoff between features and quality, discuss it with them and decide together where you want to be, or whether the whole approach needs to change (e.g. by hiring a tester). If your PM does not understand this, then you have a lot of explaining to do...

  • 9
    "If your PM does not understand this, then you have a lot of explaining to do..." Story of my life. – Captain Hypertext Feb 8 '17 at 4:21
  • 2
    How many millions is the client ready to pay to ensure there are no bugs? – David Feb 8 '17 at 4:59
  • 2
    @David For a website? I don't think even "millions" would quite cover it... – Luaan Feb 8 '17 at 8:50
17

Background: Software developer on PC's and devices for 30 years.

No. It's not normal to expect no bugs

I would extend that to say it's not reasonable. Bugs are to be expected. I am most afraid when I write something complex and it has no bugs. That usually tells me that I'm missing something major and I question whether it's even hitting the methods it's supposed to.

There will be bugs. Period. No software except the most simple mundane program is bug free. None. Ever.

I would ask your PM to find you a company that has never needed to patch its software. Zero bugs is a goal. Nothing more.

Allow me to add this however. None of this means that a bug in production is acceptable. Fix it.

  • 8
    Well said. Your last sentence may seem contradictory, but it's not. YES, bugs will slip through, but when found, they need to be fixed ASAP. "I am most afraid when I write something complex and it has no bugs." indeed! I would also add that the PM's anger is misplaced. You find a bug, you fix it. Done. Don't get mad, just fix it. – Retired Codger Feb 7 '17 at 18:54
  • 3
    I'd put a bit of a different slant on this. Sure the PM wants no bugs. The PM certainly isn't going to say that n bugs are acceptable. What is stupid is that the PM gets mad. Getting mad is counter-productive. So for any bug you have to decide is it (1) critical and needs to get fixed before release (2) non-critical and fixed in a future release or (3) trivial and it will never get fixed. // A trivial bug might be using the British "colour" instead of the American "color" in some text on some webpage. Do you really need two versions of the program, or can Americans live with "colour"? – MaxW Feb 7 '17 at 21:29
  • You need to realize that end users are also amazing with their abilities to do things that are so unthinkable that it's hard to test for it. Once had a complaint that something was 1 pixel off if you zoomed into max. – Snowlockk Feb 8 '17 at 9:12
  • I remember getting griped at because a user tried dragging an image into a text field and it crashed. I shot back, "What will a doctor say if you complain that your butt hurts because stuck a grapefruit in your rectum!? He'd say 'Why would you even think to do that and just don't do that!" But some people get all bent out of shape because you can't predict edge cases, regardless of how many times you explain what "edge case" means. – Chris E Feb 8 '17 at 18:59
16

Yes, I sell custom made bug-free financial systems which just work perfectly within their well defined parameters once the customer has them. Every single thing in them is tested at least twice most are tested a lot more than twice.

It costs me a lot to make them that way, but that cost is passed on to the customer and the time to make them that way is factored in to the timeframes.

I can't understand these other answers saying it's not possible particularly for small projects. In fact not only is it possible, but it's the desirable outcome. If a customer came to me with a bug they found I'd be sacking someone. Downtime on these systems can cost customers hundreds per minute.

  • 19
    The trick is what the "well defined parameters" are... If you want a program that runs a specific function on a specific machine, then sure, bug free is easily within the realm of possibility. If you want a program that can run on any computer anywhere... then there will be bugs. Not because the software is hard to write, but because I can't test every combination of hardware and software that exists, and sometimes, it's the hardware's fault. – ArmanX Feb 7 '17 at 22:23
  • 20
    I'd be very worried about a PM who, when building something like aircraft control software, thought any bugs were acceptable. I'd also worry about a PM who demanded 100% bug-free at any cost on a kids' phone game. Context is everything. @Justin maybe only 20% of us work on stuff where bugs actually have major consequences? – Julia Hayward Feb 7 '17 at 22:33
  • 17
    In practice it's not possible to remove all bugs in a sufficiently complex system. Depending on how you define "bug" - even the test cases could have bugs that lead to them missing undesirable behavior. Aircrafts have bugs, rocket controllers, medical equipment, etc. – Chris Feb 7 '17 at 22:55
  • 2
    Found the salesman. – Cypher Feb 8 '17 at 17:23
  • 1
    Easy to make a bold statement like that, hard to show it's actually true. I don't believe a word about you selling bug-free software. You said you're not an engineer, how can you claim to have the expertise to say your software is bug-free? – Pieter B Feb 9 '17 at 11:25
8

Personally I think it's unreasonable to expect zero bugs to show up in production when there's no solid testing framework. When testing by hand, you're prone to miss errors here and there.

I'd recommend that you bring up the idea of using automated tests, for instance using Selenium or GhostInspector, to ensure that your features continue to work, even after changes have been made. Once you have these tests set up and make sure that they run before the changes make it out to production, having a production environment that's free of simple bugs should be possible.

  • I am a firm believer that you can't inspect quality into a product. Bug testing is essentially inspecting. – MaxW Feb 7 '17 at 21:16
  • This is the only practical answer. While the other answers are correct that the manager needs to understand the time-investment needed for testing, automated testing is the only way to prevent it from becoming a full-time manual job. This solves both problems. – user30031 Feb 7 '17 at 23:11
  • 1
    @MaxW this question isn't about inspecting quality into a mess, this is about catching "a very rare occasion [when] something small will slip through". Is inspecting still not applicable in that scenario? – TessellatingHeckler Feb 8 '17 at 2:54
3

Critical, show-stopping bugs are one thing. You should always try to test for those as much as possible. And there's will always be a slim chance that some really weird edge case will occur. Best way to handle that is patch it up, create a test for it, and move on.

However, I think it's practically impossible to do what they are demanding. Most important thing to me is that there's no definition of what is considered a "bug."

Is there a styling that they don't like the look of? Is the user interface not exactly doing what they expected?

That's when you need to really examine: is this a bug or an unintended outcome? For example, if I made the text blue with no requirement whatsoever saying that it had to be blue or not be red or be any color, then the project manager yells at me because it looks better in green (and there was no requirement for green), I would argue that that is not a bug and that I couldn't have been expected to know that. Make sure when they say it's a bug: it is something that is a substantially unexpected behavior when something was explicitly expected.

Now I'm not saying that you need to get really nitpicky with what is and isn't a bug. The last thing you want is to start butting heads with your project manager over such a thing. What you want to do is get on the same page on things: acknowledge that you agree that there should ideally be zero bugs, but you cannot go in expecting that it will never ever happen. It's impossible to know the infinite combinations of actions and expectations to create all the tests for them. And bugs will almost certainly happen in many different degrees. What's important is that you address them professionally and timely while minimizing user impact.

Also, maybe unrelated to the main question, but I think could be helpful: when I have to deal with someone (a manager, client, user, whatever) that is angry at me: I acknowledge their concerns, focus on figuring out what what their problem is, and respond with empathy and logic. Determine what they are upset about, do (or in some cases not do) what needs to be done to resolve it, then talk about it. Stay calm, think, and respond. Talking and communicating are often overlooked and needed when both parties need to trust in each other to get things done.

  • 2
    I think you need to look at not just whether the bug is a showstopper but whether it is likely to occur. A show stopping bug that will occur once in 100 years shouldn't be fixed for many systems. – sixtyfootersdude Feb 7 '17 at 23:21
  • Too true, @sixtyfootersdude! We should also make sure we consider the risks. A one in 100 years bug that breaks Facebook can be overlooked more easily, but a one in 100 years bug for a life critical system like for life support systems in hospitals or self-driving cars... that's a whole other can of worms we'll be getting into! – JayNCoke Feb 8 '17 at 0:35
  • From liability point of view, sure (because for some reason, people expect machines to be not only better than humans, but perfect - and their makers (newsflash - humans) to take full responsibility for error). But don't forget that every trade-off has its good side and a bad side. What if fixing the 1 in 100 years bug in a life support system means that only a tenth of the hospitals can afford the software? It's the kind of calculation that humans are incredibly uncomfortable with, because we always try to pretend that human life is priceless - prohibiting any consistent calculations. – Luaan Feb 8 '17 at 8:59
3

To add on to what others have said here, there are a couple things that I see here that are potential pitfalls for you, pitfalls that probably helped lead to a PM actually asking you to develop "bug free" software (whatever that even means), but pitfalls that stand out on their own.

  • Lack of a separate testing team. I'm a web developer too and I've been on teams that do their own testing and teams that have a dedicated QA team to do this for them, and let me tell you something: the latter set-up is far, far preferred. As a dev, you're just plain not going to be good at catching your own mistakes, particularly if you're writing against a deadline, and particularly on top of that if you are writing one very specific enhancement to a page or what have you. Even on those teams that did things "ourselves", we tended not to test our own stuff; instead, we tested each others' environments.

  • Lack of specific design documentation. I don't think "bug free" design is compatible at all with Agile techniques because the whole entire point of Agile is that you gather requirements, build something you think is going to meet those requirements, and in a week or two weeks you talk to your stakeholders about how well that does the trick. A widget that doesn't quite do what your stakeholders were envisioning is kind of a bug (and it's part of why I don't quite get what "bug free" means) and the only way to get around that in my book is to require intricate and comprehensive design documentation that is practically an outline for the web application itself.

  • Lack of a test driven development paradigm. Admittedly, TDD is a thing I have a problem implementing, especially in the world of web dev. Still, it's a good idea to do what you can to acquaint yourself with testing software that's out there, design tests that will not succeed until you're done with your enhancement, and then write to succeed them. If you're continually running these tests on your web app, and you keep the old ones around to ensure that everything still works, this potentially automates a lot of the work that a separate testing team might do, and more importantly the fact that the exact same things get tested the exact same way over and over again means that there is little room for something getting ignored.

Anyway, whatever the case may be, it really sounds to me like your PM is cutting two legs from the project triangle out from under you (in this case, he's requiring some very specific product-related issues while also not paying for people to QA your work) and in their way setting you up for failure. No, this is not OK.

3

I'm a software tester for a large engineering firm and can't stress enough how unreasonable your boss is.

Testing's fundamental aim isn't to find issues but to reduce the risk of a product. This can be done by searching for and finding issues, however to find every possible issue, according to the International Software Testing Qualifications Board (ISTQB) is impossible. It's all well and good having use/acceptance cases that work however you would be amazed at how customers will use it. They don't have a background knowledge of the system so will use intuition which will lead them to issues that you WILL miss.

Testing is a continual process that aims to reduce risk of the system and only stops when the agreed level of risk is achieved. This should be set out in a test plan when the project is started and agreed with all parties whom have an interest in the project.

Therefore your PM needs to understand that nothing is risk free and to ensure something is bug free is impossible, to get as close to bug free would require infinite testing.

He needs to agree to what extent testing will reach and set it out in the project test plan.

Search ISTQB test plan for a template.

2

This is a typical example of the cost/benefit tradeoff. You can provide a higher uptime (ie: less bugs) but it is going to cost more.

The more effort you spend on design/test/automation/etc, the less bugs you will have. It is your job to educate the product manager on this trade off and it is his role decide how much time/money/effort he wants to invest in reducing bugs.

It is your job to explain to the product manager how much extra time/effort it would require to have less bugs and then he can determine whether it is in the business's interest to do so.

I think the next logical question that should follow is: how many bugs or how much downtime can you tolerate a year? This is typically measured in nines.

  • 3 nines ie 99.9% availability means you will have 9 hours of downtime per year
  • 4 nines ie 99.99% availability means you will have 50 minutes of downtime per year
  • 5 nines ie 99.999% availability means you will have 5 minutes of downtime a year

Source

Another interesting article on uptime and whether you should aim for 5 9s of uptime: http://www.continuitycentral.com/feature0267.htm


Edit: One other thought. The desire to be bug free is not good enough in and of itself. Even NASA who aims to have 100% uptime and zero bugs is unable to accomplish it. It takes time and experience with your current environment, customers and problems before you can understand the edge cases that you will encounter and how to resolve them.

Not the answer you're looking for? Browse other questions tagged or ask your own question.