52

I manage 20 software engineers divided into 4 sub teams. Every team has good work standards and a high-level of ownership except one. That team has one senior guy and three juniors. Every time there is a critical bug (impacting the business), this senior guy always pushes the work to the next day by saying things like "I can't finish it today," "I will look into it tomorrow," "Do we really need it today?," or "How are we going to test that tonight?" Even when I told him I needed it now, he said he had something else to do and sneaked off when I was not there. He also told these juniors to push back their work as well.

Last week, I told them in a team meeting that I expect a higher level of ownership. If they promise something, they should do it. If there is a critical bug, they must fix it even if they have to stay late.

Today, there was a critical bug and this senior guy said the same thing again - "I can't finish it today. I have a meeting with friends and I have to go." then he sneaked out while I was talking to my manager.

This is not the mentality I want my team to have. I plan to tell him that he has to change his work style or find a new job, and waited for the answer. Is it too direct to do that? Is there an alternative way to deal with issues like this?

Update

In this particular example, the bug prevents 90+% of users from logging into the system. On average, this happens once a month this year while it happened twice last year. Critical bugs are well defined bugs which: 1) prevent users from logging into the system and 2) prevent users from purchasing products — only these two type of bugs.

What we did to prepare every release:

  1. We had thorough plans where everyone understand the requirements. We actually plan about field name and functions. I implemented for all teams the rule that requirements can't change after sprint start. We also have test cases ready before sprint start.
  2. We add buffer to all tasks, let's say if we think we can finish something in 1 day, we put 1.5 days. We found that some people always underestimate tasks.
  3. First deadline was end of Jan - it is when they think they can get it done with tests. This is another rule I implemented in all teams. POs tell us what they want and we tell them how long it will take. So, I told other teams that everything would be ready by 3rd week of Feb.
  4. By the end of Jan they said all features are done with tests in test cases. We deployed them to our test environment and found a bug where user can't login. It turned out that they did not write all the tests. I asked them how long it would take to fix the bugs and write the tests, they said two weeks.
  5. First two weeks of Feb, I told everyone that we would only test and fix critical bugs in these two weeks. Again, critical bugs are either 1. users can't login or 2. users can't purchase products in app. Everything else will be in our backlog.
  6. Week 3-4 of Feb after we released it to customers. We spent this two weeks fixing non-critical bugs (that we log from #4) which are reproducible crashes and other less important bugs like layout and etc. Again, all these fixes have tests.
  7. We released it to customers with all tests green. After deployment, we found that some numbers are off so we retested everything and found the same issue coming back - users can't login.
  8. Last time they stayed late at night, I gave them extra 2 days off.
55
  • 40
    Can I suggest that everybody rethinks their answers in the light of the above edits. Commented Mar 15, 2019 at 13:25
  • 35
    Are you telling me (on average), once a month, 90% of your user base is not able to use your service (during the day)?. Boy, you have to have bad QA. You must review your processes ASAP!
    – Marcel
    Commented Mar 15, 2019 at 15:18
  • 24
    That's one of your problems. Developers do not make good testers. joelonsoftware.com/2000/04/30/… If instead of 20 devs you had 15 devs and 5 testers, your product would be in better shape and you'd be paying less salary.
    – 17 of 26
    Commented Mar 15, 2019 at 16:45
  • 39
    "they must fix it even if they have to stay late." uhh oh nope you're dead wrong
    – user86742
    Commented Mar 15, 2019 at 17:42
  • 18
    If there is a critical bug, they must fix it even if they have to stay late. Is this a joke OP? Be less of a slave driver and more of a leader. RE: Last time they stayed late at night, I gave them extra 2 days off. This is also just stupid. Have you considered that 2 extra days don't matter if they had something important you made them miss on the one day they stayed late? Are you aware how bad sleep and burnout accumulate doing this? Their obligation to you is limited, you are using ownership as some sort of culty weapon to control them and it appears the senior sees right through it.
    – CL40
    Commented Mar 16, 2019 at 3:03

21 Answers 21

342

You seem to be confusing two things:

  • Them working any amount of hours to meet unexpected or unplanned issues.

  • Them being responsible and providing quality work in a predictable way.

Ownership is not about the team working the whole night to fit your promises to customers. Ownership is about knowing what's in the code, how it works, having a plan and being able to tell you how and when things will be done. Ownership is developers making the right decisions so the code works correclty not just tonight, but in the years to come.

Sorry if this is a bit rough, but I've had too many managers tell me variations off your post. More often than not it boils to:

  • lack of clear mandate
  • changing requirements
  • short term focus
  • constant urgency

Would you please elaborate, in the question on what you, as a manager, did to prepare those releases, empower your team, and how you listened to their feedback? Then we can talk about ownership.

21
  • 173
    Well said, especially the last point. If everything is urgent, then nothing is.
    – 17 of 26
    Commented Mar 14, 2019 at 18:10
  • 18
    While I kind of agree, in principle, IMHO we're ignoring an important fact. If your manager asks you to start fix something TODAY then you DO it today. You do not postpone to tomorrow because you think it can wait, no matters what. You may not finish (because rightly you may not want to work overtime) but you start ASAP, it's not your call. It's near to insubordination and as such you risk to be disciplined. Commented Mar 14, 2019 at 18:46
  • 132
    @AdrianoRepetti I strongly disagree. Poor planning on my manager's part does not mean life-or-death urgency on my part. Yes, I do what my manager wants to the best of my ability, but I also try to keep my manager's expectations in check. If they are asking me to do something that is unreasonable, I am not going to stress myself out trying to do it.
    – David K
    Commented Mar 14, 2019 at 19:49
  • 28
    @AdrianoRepetti In the original question it mentions that the employee says that he "can't finish it today", so for all you know he does start working on it, but it isn't possible to finish until the end of the workday.
    – Maxim
    Commented Mar 14, 2019 at 20:18
  • 20
    @Chronocidal Critical bugs in production can almost always be traced back to poor planning. if a critical bug gets into production, it means that not enough time or resources in the testing phase. Critical bugs will often get to the testing phase because not enough time or resources were spent on development.
    – 17 of 26
    Commented Mar 15, 2019 at 12:46
124

Even when I told him I needed it now, he said he had something else to do and sneaked off when I was not there.

Today, there is a critical bug and this senior guy said the same thing again - "I can't finish it today. I have a meeting with friends and I have to go." then he sneaked out while I was talking to my manager.

In both of these examples, you refer to him as sneaking off, but by your own words he told you that he wasn't going to do this work and then didn't do it. Sneaking off implies he's being deceptive or dishonest, but it sounds like he's being transparent, and you ought to recognize that. I've worked with people who say they'll handle things and then disappear, and those people deserve to be fired. Someone who informs you of their bandwidth and then follows through is different entirely. This person's integrity isn't an issue; he is only unemployable if his results aren't sufficient.

Last week, I told them in a team meeting that I expect higher level of ownership. If they promise something, they should do it. If there is a critical bug, they must fix it even if they have to stay late.

This is a reasonable statement and a level of ownership that senior engineers should generally accept with some caveats:

  1. Critical bugs must actually be critical. For example, in my own career I have stayed late to fix "critical" bugs that were then not deployed into production for two months. In those cases, it was a manager freaking out about something and wanting it now instead of actually a critical bug. Of course, there have been actually critical issues as well.
  2. Staffing levels must be generally sufficient. Meeting release dates and fixing issues are important, but if we are always late because we have 3 people doing 4+ people's work, that's a different situation.

Is there an alternative way to deal with issues like this?

Some development methodologies have built-in ways to manage these issues. In Agile development, for example, sprints are ways of promising what work will be delivered. It also includes built-in ways of measuring velocity (the amount of work being accomplished) and usually goes along with software (JIRA is the most popular one I believe) that makes whether or not a team or individuals are meeting those goals. In agile development, if you need to change course mid-sprint - like take time to fix a critical bug - it reflects that you're changing the scope inherently. Normally, you take things out in order to add whatever it is that must be added. This process makes it really easy to evaluate whether "I can't get to it today" is because he's working hard on other important goals or that he is just being difficult.

IMO, it's a fantastic method of software development that unfortunately is almost never done correctly.

UPDATE: in response to the question edits, this bug is absolutely critical in nature (at most companies it'd be called a showstopper instead of critical) and should be fixed immediately. I would follow the technique I described above - taking things off of his plate in exchange for him working on it now.

It sounds like this project has been a mess and very stressful for everyone involved, but a bug that makes it so 90% of users can't log in is worth staying late for. You need to assess whether or not this employee has completely checked out (in which case you have to help him move onto other employment) or if the project has just worn him down and he needs a break.

12
  • 6
    I agree with most of your comment but I'm not sure Agile applies here. I'm assuming when the OP says "critical bug" he means something that has come up in released software that really has to be fixed right now (e.g. the recent Facebook outage... I suspect a lot of people were burning the midnight oil). It's true that Agile will let you measure impact on the schedule but OP doesn't even mention the existing work schedule.
    – DaveG
    Commented Mar 14, 2019 at 19:26
  • 59
    +1 for "Critical bugs must actually be critical." Just last week I saw a "critical" item that was ultimately ranked 6th in priority... I've learned, for better or worse, that "critical" is a word which can just be ignored.
    – zr00
    Commented Mar 14, 2019 at 20:30
  • 2
    @DaveG I don't know if they are using Agile or not; I'm recommending it as a process that makes it clear what the impact of asking for a bug today more clear to all parties. My experience is that all parties involved have a better experience when the impact of escalating a bug is more transparent.
    – dbeer
    Commented Mar 14, 2019 at 21:23
  • 11
    To add to this, most engineers are flexible about working extra hours for genuinely "critical" bugs. But OP, do you then give them hour-for-hour time off in lieu? If not, expect your staff to start working to rule as this team do, because busting a gut for the company does not in the end make any sense if the company doesn't give anything back.
    – Graham
    Commented Mar 14, 2019 at 23:34
  • 3
    I love having 5 “high”, “show-stopping” defects that boil down to “this menu is lime, but is supposed to be chartreuse”.
    – zero298
    Commented Mar 15, 2019 at 2:04
76

In my office we use to quote the following:

“Poor planning on your part does not necessitate an emergency on mine.”

In my experience developers often are motivated to help with a problem that appeared because of a mistake on their side or something unforeseen.

But all to often issues arise that are not only unsurprising but predicted. Before you decide to give your developer an ultimatum and likely make him look for a new job, you should ask yourself the following:

  • Have you done enough to avoid "critical" bugs in the first place? Did you give developers enough time to implement testing, code reviews, refactorings and monitoring?

  • Are you making sure that new features get activated when there is enough time to fix them? (as opposed to late in the evening or on a Friday).

  • If critical bugs are common, are you paying enough for overtime or on-call duty?

  • Did the developers you want to have ownership, "own" the release process? Would they able to stop a feature release, if they think it was buggy?

  • Are your deadlines realistic and agreed on with the dev team?

If all of the questions can be answered with a clear "yes", then you might have to let go of your senior developer.

If any of the answers is "No" or "I am not sure", then I would start looking for the problem in management and fix these problems first.

6
  • 2
    I have listed things I have done to prevent critical bugs but obviously it's not enough. Of course, I give them enough time because they tell me when they think they can finish it and this time includes writing tests and code review. On top of that, I add 30-40% buffer time plus another 2 week for testing. Critical bugs wasn't a common thing until lately when we had it twice in there months. And yes, the team own the release process through ci/cd. I believe the deadline is realistic because everyone agree with it from start to end of the sprint (i check in every time we have a standup meeting Commented Mar 15, 2019 at 3:24
  • > "Have you done enough to avoid "critical" bugs in the first place? Did you give developers enough time to implement testing, code reviews, refactorings and monitoring?" This strongly agree with this point. There are processes that ensure good quality software. If a manager prevents his/her engineers from completing these processes then crises that occur as a result are the manager's problem, not the engineers'. Commented Mar 15, 2019 at 6:36
  • 10
    Developer who goes "not my bug, I don't care how much the company loses because of it" is... a bad person to have on a project, no matter how technically good developer they are. Any developer who doesn't care should just find a new job where they would care and/or where emergencies don't really happen and/or the company does better job at avoiding emergencies. Staying but not caring is not good for anybody.
    – hyde
    Commented Mar 15, 2019 at 10:12
  • 13
    @hyde: On the flip side, quite some companies excel in creating such developers. And the company in the question sounds like one. You can cycle the people, but that won't solve the problem. You end up creating just more cynics.
    – MSalters
    Commented Mar 15, 2019 at 11:08
  • 24
    @CodeProject "And yes, the team own the release process through ci/cd" - that's... not what Helena means. You're talking about the team controlling the mechanism by which code gets deployed. Helena is talking about the team owning the decision about whether to release - about them being able to decide whether it's advisable to release a feature as it stands, and decide not to (and to let a deadline slip) if they think it's not ready. Your comment - which focuses most on defending the deadlines you impose on them - suggests to me that they do not in fact have such ownership.
    – Mark Amery
    Commented Mar 15, 2019 at 15:28
47

You claim lack of ownership by the team. Everything your developers build is owned by the company, not them. When you say that your employees should "own" the results of their work, does it also mean that they will receive the profits that those results make for the company? If it doesn't mean that, they don't truly own the work and you can not ask ownership from them.

If there is a critical bug, they must fix it even if they have to stay late.

Your solution to fixing critical problems by making your people stay late is convenient for the company and the employees pay the price. Again, that would be OK if they also get a share of the profits. Do they?

In this particular example, the bug prevents 90+% of users from logging in into the system. On average, this happens once a month this year while it happened twice last year.

When this happens so often and you don't install organizational procedures to reduce the impact of those errors, it is you as an organization that is at fault.

Actually, your current approach to fixing "critical" problems and your contemplation of firing your employee could be considered a sign for a dysfunctional organization. Your employee's behavior might be his way to react to that. Your update on the original question with a list of what you think you are doing right (as opposed to thinking what you might be doing wrong) also shows that you might have an issue accepting that you as a manager are a part of the problem.

There are a lot of things management can do to improve quality and reduce urgency before you ask employees to stay late:

  1. No matter how well you think that you have focus on quality, the results show that you haven't. You have to seriously improve the quality of your development process, which could mean measures like reviews, inspections, pair programming, increased testing, redesign of critical components, improved architecture and design etc. You better start analyzing the organizational issues that cause those problems instead of writing down the list of measures you have already implemented. Obviously, they are not working.
  2. Why does your employee have to stay late to fix the error? Can you do your releases in the early morning to give your developers the entire working day to fix issues?
  3. Have you thought about using feature toggles or other measures to quickly revert to the previous version of the feature to give your team time to fix the problem?
  4. You can not blame your employees for having plans for the evening when issues pop up on short notice. You can install a system of stand-by duty on days of critical releases. Then people know beforehand that they might have to stay late and can prepare accordingly.
16
  • 6
    The third point on this is very standard for critical functions on all of our workplace programs. +1
    – IT Alex
    Commented Mar 15, 2019 at 14:23
  • 3
    @rkeet: totally disagree. it's extremely important to be able to toggle on/off specific features at runtime without having to redeploy anything. And this has absolutely nothing to do with having your applications containerized or not. I don't want to have to involve thirdparties / release managers / platform supporters just to disable / enable a simple feature that's causing havoc if I can avoid it. Commented Mar 16, 2019 at 16:43
  • 1
    @rkeet - your points make sense in a server-side environment, but as has gradually come out in the comments, the question is actually about a mobile application - which is a situation where on the iOS side updates have to go through app store approval latency and on the Android side there is an untestable variety of platforms implementations, and on both where the installation of updates is subject to manual end-user approval. Feature toggles or even "this version is broken, you must update it to get to any other screen" server responses are clever attempts to deal with these realities. Commented Mar 16, 2019 at 18:13
  • 1
    @rkeet: so by your reasoning, if your right arm hurts, it must be because your whole body is damaged? If feature A is problematic (and it can even be because another system is malfunctioning and it's preferable just for safety to also disable this functionality) then you shut off feature A, you don't roll back a whole sprint worth of functionality. Commented Mar 16, 2019 at 20:17
  • 1
    @rkeet: and are customers (and other service's owners) going to be willing to have your old version running for hours (or even days) until you find the problem, fix it and properly test it? What if then it's not really fixed and you have to roll it back yet again? lol. So your plan is to have the whole office looking after you instead of properly isolating the problem (shutting it off) and calmly fix it while keeping everything else running smoothly as always? Good luck. Commented Mar 16, 2019 at 21:00
40

Working in software this is very common.

You treat your people as professionals. You're talking ownership but then giving demands that a 'critical' bug must be fixed NOW.

Is the bug actually 'critical'? Is it the result of unclear requirements? Our old friend 'scope-creep'?

In each of these you (as the manager) need to manage expectations. Not every bug is 'critical'. Requirements can suck. Project scope changes.

Instead of demanding they drop everything for something 'critical' work with your teams to when it will be fixed. Then hold them to this estimate.

I've been putting 'critical' in quotes because after 30+ years in this field (yikes I'm old) this term is very misused. Everything can not be 'critical'.

17
  • 30
    Holding people to their estimate is pointless - it's called an estimate because they don't actually know when it'll be fixed.
    – Erik
    Commented Mar 14, 2019 at 19:47
  • 26
    @JMac when it's solved they will know what was wrong and where the time went, if you want to have a retrospective. But until it's solved they can only tell you what the time has gone to trying (or what other obligations have gotten in the way), and maybe their current hunch for what to check / try next. Some discussion along the way can be productive and insight can come even from the act of conversation; but there's a point where discussion and reporting itself become a self-defeating source of delay. Commented Mar 14, 2019 at 20:20
  • 6
    Indeed, estimates on complex problems usually aren't meaningful, and everyone with any sense knows that. At best for something large with the right guess multiplying factor you may come out approximate on average, but the specific time sinks are rarely those expected, so it's really just a test of one's pessimism skill. Commented Mar 14, 2019 at 21:01
  • 6
    In that case you want to have people make an honest estimate, then multiply that by 5, and than tell you that amount of time, just to make sure that it will be likely that the issue is fixed within that timeframe. That's no longer an estimate but safe expectation management.
    – user57251
    Commented Mar 15, 2019 at 8:11
  • 2
    Here is nice talk about estimates: youtube.com/watch?v=QVBlnCTu9Ms (I don't agree in every aspect with everything, but it makes a few good points). Hint, the title is "#NoEstimates" Commented Mar 15, 2019 at 12:21
35

With the updated question, it is now clear that you are trying to fix the wrong problem. The senior engineer's behavior is a symptom of a fundamentally broken software development process and/or dysfunctional company.

If you have critical bugs getting into production every month, then you have at least one of the following problems:

  • Incompetent engineers
  • Unmaintainable code base
  • Inadequate testing

Given how much manpower you have at your disposal (20 engineers is a LOT of resources), it's likely a combination of all three.

My guess is that the senior engineer is fed up with the constant firefighting, and rightfully so.

You need to dig deeper and fix the underlying problems that are creating the need for people to continually work late. Convincing one engineer to work late more often is not going to help the big picture.

Now, what to do about it...

Step 1: Figure out why testing is not catching these critical bugs

The first thing you absolutely need to do is stop these critical bugs from ever reaching production. Every bug that reaches production is a failure in the testing process.

Go back over every critical bug that was discovered in production and determine exactly why it was not caught in testing. Add more automated test coverage, manual test coverage, or testing resources as necessary.

Step 2: Determine the root cause of every critical bug

For every critical bug, find out:

  • Who created the bug
  • When the bug was created
  • Why the code was being modified
  • Where the bug was introduced in the code

By doing this analysis, you will discover some patterns. Maybe there is one or two developers who keep introducing these bugs. Perhaps there is one code module that is very difficult to modify without causing problems. Or it's possible that the code as a whole is very difficult to with.

11
  • 22
    And STOP ADDING MORE BROKEN FEATURES until the critical bugs in the existing code are fixed!
    – shoover
    Commented Mar 15, 2019 at 15:33
  • 5
    Sometimes these issues aren't even "bugs" but rather fundamentally wrong design incompatible with the environment in which the code runs; you can fix an endless stream of "bugs" surfaced by the new way that causes breakage every time the code encounters previously unseen (but perfectly compliant) behavior of interacting systems or OS layers, or you can take time to fix the underlying design mistakes. Commented Mar 15, 2019 at 15:57
  • 2
    @ChrisStratton Exactly, which is why it is absolutely necessary to understand the root cause of these critical bugs.
    – 17 of 26
    Commented Mar 15, 2019 at 15:59
  • 2
    I very like this answer but I'd expect a SENIOR engineer to go to the manager to discuss the problem when he is "fed up" with something, to stop doing "firefighting" is not IMHO the response I want to see from a professional experienced engineer. Commented Mar 15, 2019 at 19:17
  • 6
    @AdrianoRepetti - who says they didn't? Developers usually get into this cynical phase when they learn that management in their company doesn't listen.
    – Davor
    Commented Mar 16, 2019 at 12:05
29

I want to make one additional point. Rushing out a bug fix often leads to technical debt. If your senior developer is questioning how it will be tested tonight then that is a good question that a senior developer should be asking! I’ve worked at places where urgency is prioritized over quality and this has had negative long term consequences. Ultimately, your team will have reduced capacity because it is always fighting fires.

7
  • 4
    Yes, and don't automatically assume that just because the other teams would work back and fix the bug that they're in the right on this. They may even feel the same way, but don't want to pick a fight with management over it. Commented Mar 15, 2019 at 0:52
  • 1
    @CodeProject the sequence of events in your edit is actually a pretty good example of what this response is warning about. The tests you ran in late February did not catch this issue, so no, the testing concern was not actually fixed. Likely there are areas of your codebase (or its interaction with the underlying mobile OS or remote services) which are not yet properly understood, and as a result the bug fixes continue to contain unsafe assumptions that break in situations outside those you've thought to test for. Taking time to really understand it will be needed, tests can't catch all. Commented Mar 15, 2019 at 4:04
  • 13
    If you really want "ownership" you are likely going to have to be open to letting your senior people determine more of the agenda in order to include the things they need to do to really get a handle on the underlying issues. In contrast, if you dictate the goals to the degree where they can only address symptoms then in actuality you have taken ownership in a way that precludes them from having an opportunity to do so. Commented Mar 15, 2019 at 4:36
  • 1
    @ChrisStratton I agree with the part that we need to take time to understand our code base. For the second comment, we actually had 2 sprints that they decide what they wanted to work on which were refactor code, fix bugs, add more tests. Any thoughts on how to tackle this issue? Commented Mar 15, 2019 at 7:05
  • 3
    Rushing a fix can also have catastrophic short term consequences,
    – gnasher729
    Commented Mar 15, 2019 at 21:14
23

It sounds like you have a huge testing problem. You ask why does everyone not drop all outside commitments to put out a fire but the real question is why are there fires starting every month?

Do you have any QA/Testing? Why did they not find that the first and most basic step (logging in) does not work. How did something that does not work at all get pushed to production.

Also why is your response to users not being able to log in to get everyone to stay late rushing "critical" fixes instead of having a system admin revert the update and the update can be attempted again later after the issues have been fixed.

"How are we going to test that tonight?" This is the correct response. When there is a critical issue and you are being pressured to fix it right now how will developers set aside time to properly review the changes are correct/high quality and how is QA meant to check that everything else is still working after the change. It sounds like you are also asking for these changes at the end of the day where everyone is tired and their thinking ability is at its lowest making it even more likely other issues will sneak in to this critical fix.

10
  • 3
    The question and its updates make it clear that there is testing and that the testing was critical to the release decision - but also that the testing that there is, is not catching the flaws. Some modern environments have enough distinct moving parts that expecting tests to catch everything is naive, since unsafe code can work or break depending on conditions outside of the test environment. What that points to is a system with aspects that no person on the team fully understands. Commented Mar 15, 2019 at 5:15
  • 5
    @ChrisStratton If your testers are unable to actually test things then I would suggest that that is a problem itself. That also does not explain why there is no process in place to simply roll back changes that went bad. The developers likely have no control over the testing and ops procedures and are sick of constantly having to deal with failures in the process.
    – Qwertie
    Commented Mar 15, 2019 at 5:18
  • 5
    No, neither testers nor automated test harnesses can cover every eventuality. They have their role, but the set of possible interactions is larger than you can enumerate and easily involves more physical variety than you can either reasonably purchase or simulate to include in your test coverage. Patching what broke the tests last time is not enough - it has to be right by design, and not have fundamentally unsafe parts that just happen to work in all the tests tried. Software deployed to customers is not necessarily as easy to just roll back as something server side. Commented Mar 15, 2019 at 5:30
  • 12
    @ChrisStratton Both of those are true, but they should be known and accounted for. If you are unable to get sufficient test coverage, and you are not able to rollback the deployment then you should arrange in advance for a developer to be on hand to cope with any issues that may arise. I react much better to 'Can you work late next friday to cover any issues arising from the release?', than 'Its all gone to hell again, cancel your plans'.
    – Phil
    Commented Mar 15, 2019 at 12:11
  • 2
    @ChrisStratton reading OP's further comments it sounds like their idea of testing is limited to having the developers write unit tests.
    – Aaron F
    Commented Mar 15, 2019 at 17:59
10

When are people most productive? When is the team most able to handle critical bugs? There have been studies that answer said questions to when humans are best able to handle certain tasks.

You have a critical bug, and you want, a) Sr. to switch mental gears, b) Pick up a new "critical" task, c) work "till whenever" to fix it. And you expect this critical patch to work? Honestly, what do you expect for the product, the team, the team members if your wants were satisified?

Let go of your ego, and your irrational beliefs.

4
  • 3
    So it's late in the day and you find a bug where 90% of users can't log into PROD... you are saying that if studies show that your best work is done at 10am in the morning that you should wait until then to work on this? That doesn't sound silly to you?
    – JeffC
    Commented Mar 15, 2019 at 16:50
  • 4
    @Jeffc If 90% of users can not log in, why was this not picked up during testing before rollout? That is not a bug, that is a system error. Where is the dedicated support team out of the four teams mentioned above? Even if rotating.
    – paulj
    Commented Mar 15, 2019 at 17:24
  • I don't disagree with your comment... but I do disagree with your answer. Either way, the point is when the PROD issue is found is the time to fix it, not to wait until the optimal, most productive time for the employees.
    – JeffC
    Commented Mar 16, 2019 at 0:39
  • 2
    @JeffC I would say it's time to roll back from the experimental branch they should be deploying this code on. Then methodically fix the issue without rushing and possibly introducing other bugs
    – user87779
    Commented Mar 16, 2019 at 17:24
10

The term you are looking for is Discretionary Effort not Ownership.

I am assuming that your employees are meeting their contractual obligations (otherwise your course of action is clear).

You have no right to expect discretionary effort. That is what it is by definition. Fundamentally this is not something that you can speak to them about and expect a change. You are likely to get the opposite response. They are under no obligation to give it. Threats about firing them are likely to have such an overwhelming poor response, as well as being illegal.

I don't have any good suggestions on how you can improve things. The very fact that you can rely on Discretionary Effort by some of your people suggests to me the culture is not necessarily broken.

Fixing this will take time, so instead, I can offer stop-gap measures:

Fix the bus-factor of 1

Why can only a single employee resolve this issue?

Have an on-call roster

According to reimbursement agreed upon with individual employees, not what you think it is worth.

Roll out updates at better times

It may not be possible, but rolling things out at better times can increase the chance for someone to assist.

The worth of your software is a function of how well it is supported, so you shouldn't use Discretionary Effort as a crutch. If you want your software to be supported to a level, you need to ensure you have things in place to ensure it.

11
  • You missed the part where the OP said the senior told the team to go home. Bus-factor > 1. Also, I'm curious about this whole "discretionary effort" thing. Whatever term you choose to use, in the US, engineers are professionals, meaning they essentially get paid to do a job, not to work for X hours (they decide, or in most cases give up the right to decide) how long things should take. The engineers delivered a product that they greenlit, which turned out to be defective. If that is the case, I think an employer legally has the right to expect OT, even unpaid OT...
    – Mars
    Commented Mar 15, 2019 at 9:23
  • 3
    I can't speak for the US, but in Australia "workers" fall into two categories, employees and contractors. Engineers may be employed as either. Most of the time, workers are employees. To the best of my knowledge, every single engineer (200+) I've worked with has been an employee. Commented Mar 15, 2019 at 9:33
  • 3
    Regarding unpaid overtime, in Australia, forcing an employee to work unpaid overtime, even if they made an error, would be considered illegal. Commented Mar 15, 2019 at 9:34
  • 1
    I think the two terms that seperate them in the US are "contractual engineers" and "salaried engineers". You may very well be correct that most/all engineers are contractual engineers. (Also something that may be interesting is the legal definition of the term "engineer" - in Europe it has a specific well defined meaning - this may not be true for USA and "anyone" can be an engineer?) Commented Mar 15, 2019 at 9:38
  • 1
    The legal meaning of "engineer" varies by state in the US. Regarding software engineers, it's very common for us to be either contractors or employees in the US--and I've recently worked with an Australia-based contractor. (The contract involved still wouldn't permit "forcing" to work overtime.) Commented Mar 15, 2019 at 10:36
8

So, you expect your employees to give up their social and/or family lives at the drop of a hat in order to fix problems?

Are they really all that critical?

Managers always seem to think that everything is critical because saying no is hard. This is a strong potential reason why your lead dev is pushing back. They are trying to protect their boundaries because you won't. And they are trying to protect their team's boundaries because you won't.

If they truly are all that critical, then what is going wrong that allows these issues to happen?

If your product quality is that bad, then you need to move over and let your developers devise a plan to get the product back on track. Poor quality isn't just about bugs. Poor quality derails predictability. If you are consistently going off plan because your quality is this bad, then fix your quality. And you don't fix it by asking developers to do it in their personal time. If that is the expectation you set, then you are telling your developers the business does not care about quality and therefore does not value predictability. If you do not value predictability, then stop complaining.

If they truly are all critical, then why don't you plan an on-call rotation?

Not only does this protect employees' personal time and protect the business's needs, it also creates incentive for developers to fix the systemic problems that are causing them to fire fight so much. (maybe you need more or better tests, maybe you have broken legacy code, etc.)

Why don't you stay late and fix things?

You're complaining that somebody doesn't step up to work through the night to fix a problem. Why don't you work through the night to fix it? I think you'll find the same conclusions as your team lead.

Your behavior

You have threatened to fire your employees for not doing something which you yourself refuse to do. You are complaining this happens a lot, yet you have not planned for it with an on-call rotation or by repaying technical debt.

Reading your list of steps to plan a release, what stands out to me is the frequent use of "I told them to..." and the granularity of planning all the way down to function names. You plan out minor details that are easily changeable, but won't plan a support process for your product.

This is 100% your problem.

Your team

It sounds to me like you have a bunch of smart, honest, professionals who know how to make good software, but their manager likes to dictate to them how to do their job and when the manager's approach causes a problem, force them to work more hours.

Have you stepped back and asked your team how to get less critical bugs? Have you asked your team how they think they should handle responsibility for unexpected critical issues?

Your team lead is right to push back on your expectations. And I'm glad to hear that he is encouraging his team to say no to things. He is trying to protect the team because you aren't.

In my time as a team lead, I can tell you that one of the hardest but most important lessons is learning how to say no. Maybe you can learn something from this employee of yours.

17
  • 2
    If the problem was lack of testing, the solution is to test more. Not force your developers to work overtime. If you choose to not test enough, then you will always have critical issues and your developers will always have to work late. Your lead dev sees it and is trying to avoid setting a precedent of "Well, last time we didn't test, you saved us Joe, so...". Talented developers quickly turn into super heroes and it is up to them to protect their boundaries.
    – Brandon
    Commented Mar 15, 2019 at 16:54
  • 2
    Points 3 and 4 are in conflict. You're saying you can't fix the issue after hours because of the App Store limitations, but then you say you fixed it after hours.
    – Brandon
    Commented Mar 15, 2019 at 16:59
  • 2
    @Brandon - no, the app stores don't prevent creating a fix, but they make an instant rollback and instant update challenging and potentially impossible. While I disagree with most of what the asker is saying, going to bed with the feeling that a problem is solved has advantages, even if deploying the solution may have several days of external store review latency. But late night fixes tend to address symptoms and simple mistakes, not root causes. Late night understandings of deep problems can be rewarding but will still need a lot of effort to become solutions. Commented Mar 15, 2019 at 17:34
  • 2
    @Brandon I wouldn't say it's lack of testing. We usually spend 1:1 dev:test but in this case we spent 1:1.5 dev:test. That's right, we know we need more time to test. I look at it as they said the product is ready and it is not, so they need to keep their words. Commented Mar 16, 2019 at 7:03
  • 3
    "I look at it as they said the product is ready and it is not, so they need to keep their words." That attitude will be the death of your relationship with your people. Commented Mar 16, 2019 at 14:22
4

You can't force someone to do overtime (depends on country and potentially extreme circumstances as exeption in laws)

If the person can't or won't do it, it is your responsibility to find a willing employee or to hire external help if the task is vital and needs immediate attention.

Ask an employment lawyer in your jurisdiction to clarify.

As for ownership and following through with assigned or promised tasks, you have your disciplinary arsenal all the way up to ending employment contracts.

Also, what Sefe said...

13
  • 2
    Very good answer, and I think it cuts to what I think the very crux of the issue is. In addition, if the OP wants guarantees, they should hire contractors, who are not paid a salary. Contractors would be obliged to deliver no matter what (however they see fit). Commented Mar 15, 2019 at 9:18
  • 2
    Also, just want to say, an employee not following through with an assigned/promised task is likely to be a performance issue, rather than a disciplinary issue, assuming the employee worked on it to the best of their ability. Commented Mar 15, 2019 at 9:24
  • 1
    @GregoryCurrie agreed,it could become disciplinary if it's done on purpose. Commented Mar 15, 2019 at 9:43
  • 1
    Nobody capable does mobile app development on a fixed price contract, precisely because of issues like these. Commented Mar 16, 2019 at 14:27
  • 2
    In contrast, when people who actually understand the subject matter work together - as a manager and employee, or as contractor and client, they understand where the the challenges and risks are, they understand that some cannot be predicted, they understand that the writable specifications never capture the full detail of the need, and so they work together to solve the problems, rather than burn the relationship making unfulfillable demands. Commented Mar 16, 2019 at 15:56
4

Answering the updated question:

Your big problem is not a lack of ownership. This rather seems to be a sympton of deeper, underlying problem: The fact that your Development Process seems to be substantially broken.

In theory your process (i.e. automated tests & test coverage, planning in sprints, no sudden requirement changes) should prevent most of the issues you see.

By your own statement, you ran into multiple "Showstopper" issues with the program even when deployed to the customer and some of them even being regressions. The sprints are not finished on time (tests are part of the sprint). And even when written the test do not provide sufficient coverage to catch those bugs. You also said you already had a "stay late" situation (for which you gave them days off afterwards).

You need to discover what is going wrong. Only by finding and resolving the underlying issue you can hope to fix things.

1
  • It's not even a coverage issue... They reran the tests and the errors are coming out again. In other words, someone screwed up the tests OR fixes caused regression! So who is responsible for failed tests? The senior who greenlit it or the manager who believed the senior?
    – Mars
    Commented Mar 16, 2019 at 4:32
4

If 90% of users cannot log in, and users are not able to make purchases ( i.e. sales are being lost ) you need to revert the update to the previous working version immediately. Waiting for your developers to troubleshoot and fix the bug can take much longer and cause more of a negative user impact than simply reverting to a previous version.

More importantly, your developers are less likely to want to continue working for you if they are forced to perform overtime work when there is a better solution available. If you value your employees you should respect their time outside of work.

2
  • If it was websites or backend services then we would have rolled it back. Unfortunately, this is app on AppStore. Commented Mar 16, 2019 at 7:07
  • 1
    @CodeProject There are limitations, but there are ways to speed up the rollback (actually just an update) of your apps in the store, if you haven't done so already!
    – Mars
    Commented Mar 16, 2019 at 7:48
3

Answering the updated question,

It appears good working practice has been followed:

  1. sprints
  2. feature lock down close to deployment
  3. time estimation from developers
  4. automated tests

I partially agree there is a positive culture of fixing critical bugs in overtime. However the culture also needs to reflect no code is perfect, unless you spend a lot of money on it, there's an age old storey saying that NASA spent around $1,000 per line of code!. I won't comment to much on the cultural side, as that has already been covered by others, instead some methodology suggestions:

A team structure in fashion at the moment is feature Squads who own end-to-end delivery and operational responsibility for isolated vertical they write. If it goes wrong, they are the ones woken in the night, however I'd be wary in introducing this in a legacy environment as it will heed distaste if a team didn't have the opportunity to introduce the quality they desire from the get-go. And traditional Ops roles with contracts/pay to suit are likely more open to on-call way of working.

A better idea would be to introduce the idea of QA champions within a team and the "3 amigos" approach i.e. product, developer and QA member all write the Feature Spec (Behaviour Driven Design) together. This ensures that "how a QA member would break it" is accounted for from the beginning and it should be in just enough detail i.e. a specification by example manner. The QA member doesn't need to be the person writing the automation, but they should code review it. As people have mentioned above the writer of the code shouldn't be solely responsible for accrediting it's quality and introducing a 3rd party as early in the process as possible is a positive move.

Perhaps also the Production environment and release management needs enhancing. "Blue/Green deployments" and "Testing in Production" are common practices for gradually rolling out changes to a wider and wider audience only as metrics prove themselves. Ideally your staging environment should catch critical bugs, but there's always something different about production, therefore it should never be a big-bang release.

Judging from the timeframe although you are using a typical release cadence, you may wish to consider releasing more often. More frequent smaller releases can lead to less risk if coupled with good test and release automation. This can be paired with feature switches so that features can be switched on in full when the composing user stories are complete.

2

I read your update. Your problem is that you shipped a broken product. That’s it. That’s what you need to work on - don’t ship broken products.

Your complaint is ridiculous. YOU decided to ship a product when it was broken. That’s where the buck stops, with you.

You then made two mistakes: First, it seems that the timing of your release was such that you were told about it at 4pm. Release half an hour before developers arrive at work. Very simple solution, it’s your job to know this. Second, there seems to be no motivation for the employee to work overtime. Paying for overtime usually works quite well. An environment where you allow broken code to ship regularly doesn’t.

5
  • 2
    What? It sounds like OP deployed a product that the senior marked as all green. The OP may not even have a technical background (although in this case it sounds like they do), but the senior engineer is the one responsible here... OP might literally not even know what some of the tests mean, but the person who is paid to know what they mean said they were fine, then, after release, said "ah, actually.....!" OP decided to ship a "working" product, not a broken product.
    – Mars
    Commented Mar 16, 2019 at 4:19
  • 1
    "There seems to be no motivation for the employee to work overtime" Um... it's clear that that's about to be a condition for keeping their job! It's also not stated that it's unpaid OT, and it's explicitly stated that last time they worked a little extra, they got a LOT of extra time off
    – Mars
    Commented Mar 16, 2019 at 4:23
  • @Mars Apple releases my apps on the AppStore when i press the button. Totally under my control. I can also release to a small percentage of customers first. Play Store does the same.
    – gnasher729
    Commented Mar 16, 2019 at 15:07
  • My mistake, I was not aware of manual release! But the other points still stand. In addition, assuming that the release was around 4pm is mostly projection--releasing at 9 and getting reports at 10 still only leaves 7 hours to diagnose, fix and retest. If it takes more than 7, then that means OT...
    – Mars
    Commented Mar 17, 2019 at 2:31
  • Also, OP may not be responsible for the release--that could be the client. In that case, they need to discuss with the client about using that feature, but even bringing up the subject will likely not go over well with a less than tech-savvy client... ("What do you mean delay the release? You said you tested t? We paid you to test it. If you can't have it ready for a full release by this day, then maybe we should have someone else who can...")
    – Mars
    Commented Mar 17, 2019 at 2:35
1

In light of the updated question, I suggest that while you may have an employee attitude problem on your hands, what you really have is a software quality problem:

  1. There is no quality assurance process independent of the developers who are writing these features. Developers are notoriously bad at testing their own code, largely because they naturally start with the assumption that anything they wrote does and should work the way they intended it to, as opposed to how the end user expects it to.

  2. Whatever testing environment you have isn’t adequate for reproducing problems identified by your customers. You identified bugs that prevented login at several stages in the process, but the final check did not identify this problem.

  3. The senior developer is willing to ignore critical issues that actually are critical. “Dude, my friends and I were going to do something cool tonight” is not the kind of reaction to “nobody can log in therefore our business is losing 100% of this products’ possible revenue that we use to pay your salary” you want to see. If this is how he reacts to an emergency, how do you think he deals with minor stuff? With more attention to detail or less?

There is a specific part of your updated story that raises many more questions for me and I think you need to investigate in detail: At one point in the process, the team said they wrote all of the tests and there were no issues. Then you say later, it was discovered that not all of the tests were written after all, that the login bug was not identified, and that you devoted additional time to writing those tests and fixing the login issue, which later was demonstrated in production to be not fixed.

  1. When you said the tests were not written after all, why were they not written? Was it that the case in question was an oversight, or did the team misrepresent the completion of their work? The former is reasonable to expect will happen occasionally, the latter is unacceptable behavior that you need to make clear will not be tolerated in the future (unless it was a misunderstanding on your part).

  2. Once the tests were written, how do you know that these tests are correct? Did you review the tests? Clearly they don’t work; they all passed but the problem that was explicitly tested for and said to be fixed still made it to production. It is not unknown for unscrupulous people to write tests such that they always pass, especially if there is deadline pressure. If you are not capable of reviewing the tests yourself, you should find someone technical on another team to review the tests for you.

There’s also something else you wrote, that the number of critical bugs is now up to once a month, but before it was more like once or twice a year. Why is that? Have you done any investigation into why that is now happening? Did the product or team change significantly in that time period? This sounds like quality is slipping.

Here’s what I think you should do:

  1. Hire a good, experienced QA tester and subject all of this team’s work to independent QA testing (you should do this throughout your company, but this team in particular needs it because quality is slipping).

  2. Review the test environment and compare with production to ensure that the test environment still reflects production.

  3. Review tests and code your team is writing for quality and correctness on a regular basis and in more detail than you are currently doing.

You might have an attitude problem on your team with the senior guy, but you definitely have a quality problem. Attitude problems are hard to fix, but quality problems are easier to fix, and have the added bonus of rendering the attitude of one guy on the team irrelevant if you do it right.

0

Being a senior team member myself, and having to deal with such situations time and again, I can only tell that a guy whom you can always rely on, needs some motivation as well. The guy is surely honest and has his right to his personal life after office but he will stay and fix bugs if there's a better motivation for him, or if you manage to create a friendly environment for him.

Think of it the other way around: what if the guy gets sick or leaves the job or has some emergency and he has to leave the office for a time. Currently his time with his friends seems to be a problem for you but he can have all the legitimate needs to leave the office even during work hours (and spending tim with friends after office is a legitimate need as well).

A very common managerial attitude is to always through it on the guy who can do it. Now this varies person to person but being an employee, this can lead to frustration and such situation emerges when employee starts saying no to you.

So in my opinion you need to do two things:

1:) You should create a platform of learning for junior staff, and gradually delegate things so that in time of critical need you don't have to look at the same guy each time.

2:) As long as within office hours the guy is doing what he is supposed to do and meeting his KPIs, he is fine and needs to be dealt with in a friendly manner.

-2

It sounds to me that he simply places a higher value on being away from the office than in it.

There are many reasons why, and a good manager would not simply assume the worst. Perhaps he's having personal issues at home, be it a relationship, sickness, whatever other stresses.

Perhaps he is growing dissatisfied with the job and simply doesn't care anymore.

There are many reasons why he may be acting as he does. I'd suggest seeing if there is something that you'd want to give extra grace for before you rip into him too much, or discipline him.

1
  • Comments are not for extended discussion; this conversation has been moved to chat.
    – Neo
    Commented Mar 17, 2019 at 20:34
-5

Welcome to WorkPlace, where everything is the manager's fault. Everything.

I look at it as they said the product is ready and it is not, so they need to keep their words. – Code Project

Frame the question a different way.
You have a senior engineer who has been assigned responsibility for a feature, is responsible for the testing and has passed the final product. That green lit product has critical quality issues. (And I agree that your definition is pretty critical!)
In other words, you have an under-performing senior engineer who does not wish to do overtime in order to correct their own mistakes (or as you put it, fulfil their promises).

But the key here isn't that the engineer won't do overtime--from the comments, you have a mobile app, so it's not like you can release a fix quickly anyway.
The key issue is that the senior engineer is green-lighting these broken products in the first place.

Expectations/legalities here greatly depend on where you are located, but on a personal account, I have never worked at a company (US or Japan) where your senior engineer's lack of ownership would be accepted. Whether at-fault or not, it's a given that a critical error must be fixed as soon as possible. Luckily, I have also not worked at a company where I wouldn't at least get paid (within reason) for my efforts to fix my own mistakes.


The next step here is to review and find why these mistakes are occurring. Most answers here suggest that the problem is something systemic--I'd say the fact that only one out of your multiple teams has the problem suggests otherwise.

Still, you should review, find the cause and try to fix it. If it's merely an underperforming senior engineer, then steps need to be taken to get that engineer up-to-par or to get another engineer who can do the job properly.

14
  • 3
    The cultural differences are interesting (UK). I consider myself dedicated. I happily work odd hours to accommodate people that I work with AND actually my contract states that I should work my hours "to meet the needs of the business", but if I was told, on a moments notice, that I must work overtime to get X bug fixed, I'd be half way home before the door stopped swinging, mentally updating my CV as I went. Different story if ASKED and I made the decision to do so off my own back obviously. Fortunately, employers and clients who respect employees work-life balance exist and I got lucky. Commented Mar 15, 2019 at 13:47
  • 3
    "Welcome to WorkPlace, where everything is the manager's fault. Everything." -- Completely agree. I read the question then braced myself for the onslaught of answers declaring that OP the is worst manager in the world. Was not disappointed.
    – GHP
    Commented Mar 15, 2019 at 17:41
  • 1
    @Graham If live breaks multiple times a month, and they've not introduced measures to stop that happening, who else do you hold responsible?
    – UKMonkey
    Commented Mar 16, 2019 at 11:24
  • 1
    @UKMonkey No info here to suggest that they aren't doing something about it... Not to mention the rate of crits wasn't particularly a problem until very recently and seems to be a very localized problem. But as my answer suggests, I agree that the bigger problem is that this senior engineer is releasing the crits in the first place.
    – Mars
    Commented Mar 16, 2019 at 13:21
  • 1
    "No info here to suggest that they aren't doing something about it" sure there is - it keeps happening. The correct reaction is to see there's a problem, have a team meeting and establish why the errors are getting through and fix it. The rate is unimportant - because EVERY time there's a failure, steps should be taken to ensure it doesn't happen again;
    – UKMonkey
    Commented Mar 16, 2019 at 13:26
-20

It seems that it's time to put this whole team on notice. If they do not start meeting guidlelines, clear guidelines, they will be terminated.

You can accomplish this by establishing guidelines around bugs/issues that are reported against what they support. For example a priority 1 bug must be started work within 1 hour, progress reported to management (you) every hour after that, until bug is either fixed or a work around in place to your satisfaction. The to your satisfaction is key, and you must support them and the other teams with getting things done when the work flows past normal working hours by being present or highly available yourself.

When they bug out and go home, then they failed to meet your requirements to work on the issue and check in every hour. This is now measurable. These guidelines would have to be met by other teams as well.

Now when they fail to meet these requirements once, they go on notice, formally written notice, CC to your manager and HR that they failed to meet their obligations. Second time, you report again, inform team that a third incident is grounds for termination. Third time, fired.

Now I am guessing this senior developer has some critical knowledge that he/she assumes makes them invaluable. Nobody is that valuable and this sends that message. They are not supporting the company, the company will no longer support them.

Expect some fallout if it comes to termination. They do indeed possess knowledge that is going to take some effort to get up to speed with others. Make sure the Jr. developers know these policies, if they can start pulling the weight even if their Sr. bails on them, then they will be able to keep things going when you terminate the troublemaker.

10
  • 28
    This approach is a sure way to earn your shop a reputation as a great place for software developers to avoid. "Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done." (agilemanifesto.org) That's the way to go. Commented Mar 14, 2019 at 19:45
  • 13
    Normally, I might agree with you. However, OP's statement Every time there is a critical bug... makes it painfully clear this happens pretty often. And that tells me that the OP is quite possibly failing at their job and putting pressure in places it doesn't belong. Personally I think the OP needs to reevaluate why they are deploying code that is this buggy to production and perhaps figure out how to fix that before firing people. In other words I'm trying to understand why a place with 20 devs doesn't have proper testing or even an actual production team to handle these issues. Commented Mar 14, 2019 at 19:51
  • 18
    If you actually implement the procedure described in the second paragraph, I expect the "fallout" to consist of the other three teams quitting to get out from under the thumb of a tin-pot dictator.
    – Mark
    Commented Mar 14, 2019 at 22:07
  • 12
    Remind me to never work in a company where you have any management authority
    – Magisch
    Commented Mar 15, 2019 at 9:02
  • 5
    If you fire your entire dev team, or even threaten to do so, rest assured that nobody will be staying late to fix critical bugs.
    – Brandon
    Commented Mar 15, 2019 at 15:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .