How to improve a software product in difficult conditions

Question

[Short version]

The software department is heavily under pressure for bad software quality and performance and too little delivery. In addition, the CEO has gets angry easily and is killing motivation. I tried everything I could think of, but the situation keeps getting worse. Any suggestions welcome.

[Long version]

I was promoted at the start of 2020 as head of (software) development in a mid-size German company. Software constitutes about 40% of the workforce. We are delivering a product which is used by all other departments (sales, project delivery, etc...).

However, our software has major issues: bad software quality, not so good code quality, bad performance, we deliver too little too late. Also, our developers are not very experienced. We have a lot of pressure from different departments (product management, sales, projects, CEO) to deliver more features and fix current quality issues. I am also trying to squeeze some non-functional features (end-2-end tests, better support from our more senior developers). So the teams are heavily under pressure.

And to make matters worse, while our CEO can be understanding and can listen, he has a temper problem. He'll shout at people, either directly or through our chat system. He really kills everyone's motivation (including mine), and it's often impossible to reason with him. As a result, nobody dares tell him what's really going on, and some people started hiding work, booking on the wrong ticket on purpose, giving an estimate they know is not correct, etc... I know he's a good person at heart, and tries to do what he thinks is best, but his behaviour really does not help.

I have been trying to turn this around ever since I started, but it keeps getting worse. What I tried:

• Solving one issue at a time: we have better processes, some end-2-end tests, smaller work packages, etc...
• Pushing back on requests, especially the roadmap: it worked to a very limited extent, and we won't be able to deliver everything we need to deliver - again.
• Talking to him directly: I tried that twice. First time, he listened, agreed I was right, and it helped for 2 weeks, before he went back to his old habits. I tried again by suggesting having an external consultant, he got immediately angry and said if we listened to him and did what he said, all would be fine.
• Ask a question on stackexchange.

However, the motivation is getting worse, the stress levels are going up, and I have the impression that if we don't turn this around quickly, we will reach a point where the (technical and non-technical) debt will be overwhelming.

I would rather not quit. I really like the company and my colleagues. We have a really good bunch of people that care about their job. Also, I enjoy my job, and I don't want to quit at the first issue. Not to mention I've only done this for 10 months only, so I still have a lot to learn.

At this point, I am not sure what to do any more. Any suggestions are welcome.

Answer 1

I think the most important thing you can do is to shield your team from the CEO's temper and the competing demands of the other departments.

If a developer is working on a feature for department A, and the CEO yells at him for a different feature that he wants today, that developer is going to feel demoralised and stressed because he can't satisfy both people.

Instead, if the CEO comes to you, you can discuss the problems that changing the plan will cause, and if the CEO still wants his feature you can warn the other managers that their deadlines may slip.

If your team can't finish the work to an unrealistic deadline, that's an issue for you to discuss with the CEO and other department heads. Don't just pass the stress on to your team, deflect it back to the source. Tell them they're trying to "squeeze a quart into a pint pot" and that you can't produce a baby in 6 months just by setting a deadline and shouting at the mother. They need to either move the deadline, reduce the feature list or recruit a bigger team (and wait a year for them to train)

You should have a list of things to work on, in priority, and provide regular updates as you finish things. That way the CEO can move stuff around but it's obvious that extra items either go at the end, or push everything else back.

You can also point out that a lack of quality is the direct result of skipping testing and reviews due to time pressure. You can use the analogy of a busy chef who can cook a few meals faster by skipping the clearing up, but is soon overwhelmed by the mess. It's your job as engineering manager to balance the time spent 'cooking' and 'cleaning' so that your department runs at optimal speed.

Standing up to a bully might eventually get you fired, but when your team realise that you have been shielding them from all the stress, and building up an efficient, effective working environment, they are likely be happy to follow you to a new job where you'll all be appreciated.

Answer 2

I've managed large teams in very challenging environments where people were considered expandable, we had to satisfy stock holders, angry production managers and deal with lawsuits.

In the end, all problems and solutions are human problems and you need to address them as such.

A common pattern is that all parties involved have their own motivations, and will ignore other problems you're facing because they just want you to solve the ones they care about. Whether it's a feature, bug, timeline, quality, etc problem, you will rarely have people consider the other things you are dealing with at the same time. It's not uncommon for people to know that things are in a bad shape and willfully ignore it to not have to deal with the problem.

The solution to this is very easy: for a while, say yes to absolutely everything, but make sure it comes at a cost they need to negotiate with other stakeholders.

For example, Marketing asks you to change a deadline because they want to show the product at some trade show. You go along with it:

yes, I like the idea, I want to help you. In order to achieve it, we will need to cut x and y, or z from the product. Let me set up a meeting with the x, y and z stakeholders so you guys can decide how you want to proceed.

Or you are told that feature A is absolutely needed for the next release:

of course, I love feature A, we should add it. It will require that we don't finish feature B or C though, let me bring in Mike and Gary so you can decide with them what should be cut. Or, we could extend the deadline, should we set up a meeting to discuss the cost of pushing it for your feature?

This goes nowhere very quickly, but by doing this you are actively training people to understand that every request has a cost. You are not blocking anything, you are actively facilitating the process but anyone that wants something will soon realize that they need to negotiate with others.

I can tell you that everyone hates having to bargain with others to get their priorities addressed and the many difficult discussions that will arise will be entertaining because you are not a part of the problem in that process.

I have done this at length in the two largest companies in our field. They're multi billion dollars companies with thousands of employees and worldwide presence. Despite the complexities of these environment, it worked remarkably well.

In the end what really happens is that all stakeholders realize that the only person that knows the whole picture is you, as a team / department manager. So every compromise they have to discuss will involve you and your opinion and, ultimately people will have no choice: either trust your judgement, or constantly be at war with other stakeholders.

Answer 3

As I have most experience with SCRUM- or Kanban-like agile schemes, I'll use that nomenclature. If you are doing something else, adapt as necessary, it's the principles which count, and most of the things apply to almost any type of software development in some form or fashion.

So the teams are heavily under pressure.

This is the key red flag for me.

In my experience and opinion, in IT we have two pretty distinct realms: whatever happens within the team, and whatever happens outside the team.

First and foremost, you need to put up very clear delineations, and label everyone as either "team member", "product owner" or "stakeholder". This should be quite easy for the first and last category. For the product owner, it is possible that you have none, or multiple ones, so this could be harder. The restrictions are:

• Every team member should, if at all possible, be a member of exactly one team, not more. Sometimes this is not possible, which means additional overhead for time splitting, so it should be a high goal.
• Teams should not be too big or too small. "7 plus minus a few" is a good number. Nothing fancy here.
• For each distinct software product you have in your company, there should be exactly one team responsible for it; if a piece of software is too large, split it up along some line. Refer to Conway's Law, which works both ways.
• There must be exactly one product owner per team. Not zero, not two, certainly not more. This is arguably the most important individual person in the whole scheme, and the cardinality of this role is the single most important fact.
• You might have experienced, highly specialized senior devs which get to jump from problem to problem within the company and are too scarce to put in one single team exclusively. Ignore those for the time being; treat them more as a resource which can be used by the regular teams to get fires under control.

Secondly, implement some kind of sprint based routine if you don't have it yet. If you are doing SCRUM, then you have it anyways. If you are doing something more freeform (like Kanban or whatever mixture), then make sure you still some kind of a sprint cycle to structure your regular meetings or whatnot. This is simply to make time planning etc. easier and to reduce chaos a bit. An easy way to give some sense of security to your teams (and also to stakeholders).

Finally, you need to put up clear communication lanes:

• Stakeholders can not give orders to team members.
• Stakeholders talk with the product owner at all times for new features or changes to already planned-out features (most specifically for features within the next sprint).
• Team members must talk as freely and regularly as possible within the team (i.e., dailies). Actually take measures that this is not optional. For example, in a daily standup meeting, every member must speak in turn. In a retrospective (if you do something like this), there must be occasions where a moderator specifically asks everybody in turn to say something.
• If you need to have team members talking with stakeholders (say, to clear up requirements if the product owner cannot do it themselves for whatever reason), then unless the team member is experienced and strong, the product owner should be there as well. The result of such talks should always be documentation in your task tracking system for future sprints, not re-ordering of tasks, especially not adding tasks to the running sprint.
• It doesn't matter that much if you view product owners as part of the team or not; the important thing is that they are the only interface to the outside world.
• If two teams need to collaborate, this goes primarily through their respective product owners. Think about it like the product owners discussing an interface contract. This obviously can be much improved later one (with processes like Scrum of Scrums, SaFE, LESS, or whatever else you can think off).

If at all possible, add a coach who is not related to the development team, and certainly not part of the stakeholders (i.e., a "SCRUM master" or "agile coach"). Even if it sounds trite, a good one can really make a big difference. Their task is to keep the individual meetings etc. flowing, reminding everyone of what they must, must not, can, should do, keep things in a timely manner, moderate, coach everyone on the process, etc. It's a totally different skillset than developers or stakeholders usually have. Having an external one isn't the worst idea either so they are not bound by obligations to anybody.

Again, this sounds just like SCRUM, but do note I'm only describing the directly people-related aspects here. You can of course actually do SCRUM, but even if you do not want to be that strict, the same pretty much works with any other agile system. Note that I do not ask you to do the SCRUM "artifacts" (standup, planning, review, retrospective etc.) - whether you wish to do those exactly like that is up to you, other more free-flowing systems can be the better choice depending on many factors.

Doing all of this (and all the rest needed for agile development, i.e. having a backlog, a good task tracker, getting rid of technical debts, distributing know-how better, eventually doing DevOps, etc. etc.) should eventually take pressure off your team members so they can actually work again. Stuff may get done, and things may noticeably improve so you can enter an upwards spiral.

Finally, onto yourself: I do not envy your task - I have been in your shoes often, in some form or fashion, and your case sounds pretty desperate. Keep strong, this too will pass. But not too strong, or you will break. Do listen to yourself and your body, burnout is not a happy thing to happen, nothing in the world is worth it. Get communication traning for yourself to learn how to work with a shouting boss, whatever you can do to give yourself personally the tools you need to get through this. Learn how to recognize things you are not responsible for, and mercilessly get rid of them (learn the RACI principle by heart).

Best of luck.

Answer 4

Radical Transparency

I'm going to suggest that your CEO rages because he feels impotent. He can see that things are not working, and he knows that he has limited to no ability to change that fact. He is frustrated by his inability to will his company to success. If this is true, then I propose that the best way to calm him down is to give him total visibility into your team and process. Throw open the doors and say: "You must give me this week. Every day this week, you will spend the entire day with my team. You will go to every meeting, you will sit with every developer and manager, and you will see how the sausage is made. Push back all your other commitments."

Then, do exactly that. Invite him to your planning meetings, your status meetings, your standups (assuming you have them). Have enough developers "pair" with him so that he can see what they do all day. Tell him to ask what problems are holding them back. He doesn't have to understand how the code is written, but he has to understand the frustrations and limitations that the dev team faces on a daily basis. Only when you immerse him in the actual process can he see that there are deeply rooted issues that need to be resolved. He needs to have this interaction:

"Ok, what are you doing now?"

"Well, I'm working on this bug. See this ticket? Ok, it says when you push button X in app Y, you get this error message. So first I'm gonna open up app Y and see if I can repro it."

"Yeah, I get that. I've used app Y before. It's kinda janky and does random stuff."

Developer frowns. "Yeah, well, we tried to write some acceptance tests for it, but there's always new feature requests coming in, so... Ok. The app is loaded. Let me set up the repro case...ok, now when I click this button...boom! Yup. Bug is reproed."

"Ok, now what? How do you fix it?"

"Well, that's the hard part. Now I have to start up app Y on my machine here and attach a debugger..."

"Sounds complicated."

"Yeah, it kinda is. Anyway, there should be some unit tests which cover button X, but we never got around to that either."

"What are unit tests?"

"Well, they are very small tests which check the smallest bit of functionality in each part of the program, so you can quickly find bugs."

"So you're saying we can't quickly find this bug because you don't have these tests?"

"Yeah, that's why I have to attach a debugger. We might get lucky and I might figure it out in like 10 minutes, or I might spend the next 4 hours trying to figure it out."

"We can't spend that kind of time fixing a bug!!!"

Dev cowers in fear. "Well, sir...umm....I hate to tell you this, but...most bugs take at least that long to fix..."

"THIS IS UNACCEPTABLE!!!"

Manage Expectations

At this point, you get to explain to the CEO the concept of technical debt, and why he absolutely needs to support you in paying this down (say, 20-25% of your total resource allocation). But you also get to explain that as you pay it down, feature velocity will increase, because it's always easier to add features to a well-maintained system than a Rube Goldberg system.

After you get the CEO on your side, you can then proceed to gather all the stakeholders/product owners and review your org's product backlog together, explaining that everyone needs to agree on the importance and priority of each project that your org works on. Your team cannot magically deliver 5 projects at once. Your team is flexible and can change what they work on as business requirements demand, but this will incur a cost. At the end of the day, all the product owners can do is change the order of the backlog, not demand that everything move faster. If they want to impose a deadline, then they must accept whatever is ready by the deadline. If they want to demand a feature, then they must give up a hard deadline. And, of course, remind them that 20-25% of your team is always allocated to pay down tech debt, until it reaches sustainable levels.

Upgrade Team

You said that you have a bunch of junior engineers and you like your team. What worries me is that you didn't say: "Fortunately, I also have a handful of very experienced developers who can set best practices and mentor the juniors." You mentioned that you have senior developers, but not that they are good. At this point, you have to take a very hard look at your team and ask yourself: "Do I have any really good developers? Do I have any rock stars?" And if the answer is: "No", then that is your next order of business. You need to find one or three and hire them. In this economy, that should be easier than 9 months ago. Explain to the CEO that this is essential to improve code quality and delivery speed.

The fact that you built up massive tech debt in the first place tells me that you do not have any key, reliable, solid engineers you can lean on to lead the cleanup effort. If you are having to lead the charge on code quality yourself, as management, then that is a very bad sign. When you are hiring, you need to keep an eye out for engineers that specifically call out quality practices on their resume, like TDD, unit testing, refactoring, design reviews, etc. Anything which indicates that they are not just code cowboys trying to win high score on PRs per day. And you need to make clear that you need people who are not just great coders and experienced engineers who can define best practices and set standards for the team, but also teachers and mentors who can skill up the rest of the team in pair programming, seminars, tech talks, and the like.

You can't expect junior engineers to just magically get better on their own. That takes years. You don't have years to skill up your team. You have months at best, maybe not even that. If there are any user groups in your area which hold tech talks on tech debt/code quality/etc., pay for your team to attend, or invite speakers to your office to share principles, techniques, tools, etc. Make sure they can bring high-quality open-source repos to show and tell to illustrate what a well-maintained codebase looks like, and how you keep it that way.

Make a list of 5-10 good technical books in the language you use for production that highlights best practices for that language, as well as good testing and release practices, and create a small technical library for your team. Encourage your team to check out/read the books, and get the CEO to approve a modest training budget to cover books/seminars/in-house speakers/etc.

Often times, what you need is an engineer who has worked in a big tech company that does a lot of things right. That person will have absorbed good practices via the company culture, and can evangelize those practices throughout your team. Getting such a person might be expensive, but if they improve your whole team by 10%, then they will quickly pay for themselves (assuming you have more than 10 devs). You need an engineer that models ideal development behavior, teaches it to willing junior devs, and evangelizes it to the team on a proactive basis. Even just one good dev like this can turn your whole team around, as long as they are hungry to learn and improve.

I know this is a lot for you to demand, and many of these moves are politically risky. You have to use your own judgment to decide what you can and cannot pull off. It's a difficult position, but also one filled with possibility if you can turn things around. Good luck!

Answer 5

With that CEO at the wheel, it seems that you are stuffed (unless ...).

All that you can do is

1. try to get him to understand that you are hired for your technical expertise and he is not. If he does not like what you do, let him get another technical expert, but that is not him. Explain that he is likely to have the same discussions with your replacement as you have already had.

2. try to get him to understand that what we have here is failure to communicate. If he can agree to that, then do not discuss software quality, only how better to communicate. Remember that you speak a language (techie stuph) that he does not, so it is your responsibility to translate. Once you know how to communicate, things may go better.

3. if neither of the above apply, it's a slow motion train wreck, so grab your parachute

Answer 6

1. Get clear to the CEO that software development is not at all easy from a point and onwards(large+ solutions). If it was, everyone would be google-level. Extra investment solves this partially, the larger the solution the greater the other factors, even plain luck to hire the proper people.
2. Get an sql optimization expert person, if you do not have one. Most optimization problems lie into databases and you need a guy who can change sql code to run 100 or 400 or 1000 times faster at day #1 with 0% business knowledge. I do this my entire life.
3. Change infrastructure and technologies that matter where it is easy applicable. Got cloud server? Get a stronger one. Got a self-made queue software? Use an industry standard instead like rabbit mq. Try to get scalability at a fast and safe manner.
4. 2-3 if done correctly will give you some recognition. Then Spot the most costly parts of the solution and redesign them, one by one. You will need a senior+++ software engineer then to do effective software redesign. The sql+++ guy will be ready by then to propose new database schemas were applicable.

Answer 7

bad software quality, not so good code quality, bad performance, we deliver too little too late. Also, our developers are not very experienced.
[...]
As a result, nobody dares tell him what's really going on

It looks like you're working with bad tools to which you add additional features that don't work as designed beacuse your developers are overhelmed with current work that they are unable (and/or simply can't) spot possible issues and/or cannot fix it.

What I would suggest is to lift pressure from your team in terms of deadlines. So for example no current workload. But for CEO I would present it as making team more experienced. Going back with the software to revise it as getting to know it rather than looking places where your team need to add things.

This could improve the software (because whole team would look at code and came up with solution), make roadmap for future implementations ("we have this problem here but it won't affect anything unleast we add something"), which will help you better calculate time and manpower needed ("doing this will take 2 weeks but it's corelated with this issue that will take 4 weeks to fix") and make dev a little at ease when they will, as a team, have knowledge about software.

Right now your teams is doing, what I call "reconnaissance by fight". Devs are dropped with a task, they get enough knowledge to do the task and leave. So each dev have a small plot of knowledge about things they've done and seen but nothing more. Expecially "booking on the wrong ticket" hide from everyone possibility to drop by and help. It could be from simple things "I made that button but I had problem with X so I had to change Y" to more complicated ones like code always returning 4 because it was assumed you can only input "2x2" and "3+1".

Answer 8

Insulate the workers from the CEO.

The job of a CEO is to provide high-level direction for the company, not to manage individual workers. That's the job of a mid-level manager, like yourself. As a result, if the CEO is monitoring the individual tasks of employees and making comments that hurt morale, the obvious solution is for you is to insulate the workers from the CEO, so that he doesn't feel the need to monitor your subordinates directly. Your reports to him should contain all the information he requires, and any directives from him to your employees should pass through you. You could instruct your subordinates to refuse to answer his questions and then direct him to you, but that might result in him losing his temper and then yelling at them.

One potential way for you to do this is to begin working on implementing a formal project management framework like DSDM that includes roles for the CEO (in DSDM, that would be the Business Sponsor), and specifies processes for what documentation should be produced, who it should be produced by, and who it is intended for. You could probably sell this to the CEO as an attempt at raising both quality and productivity by implementing a more rigorous project management methodology with more formal governance processes for the team to work with.