How can I convince my teamleader that another data structure would be better? [on hold]

Question

I am a new hire and my current task is to implement an editor for some data. My teamleader had already made a proof-of-concept of this editor and laid out the data structure for it.

Currently the data structure is a one-to-many relation (with version history of old entries and draft states for new entries) all in one table.
I would like to change that to a normalized version spanning three tables. (You can see the detailed "pseudo"-structure in the edit history if interested. I cut it out to focus more on a non-technical side of my question.)

My teamleader said that, based on the fact that another piece of the software already uses the table as-is in production, he does not really want to change the structure. But he also said that if I bring him valid reason(s) for the change, we will change it.

He knows the rules of normalisation and he knows that this structure breaks them, but it is not enough for him. I've already tried to explain to him that it is easier for me to work with a normalized structure, but somehow I failed to convince him. I think he can't get my point because he does not work on this part anymore.

So: How can I convince my teamleader that another data structure would be better?

Answer 1

It's worth looking at this from the other perspective, namely, that of the team lead:

You have two different structures, one is tested a proven, presumably with an existing number of people supporting similar-structured applications. The second has been brought to you, with technical reasoning, but, crucially, without business reasoning.

Rather than not getting your point because he doesn't work in the area any more, it's possible that you're not getting his. The reasoning needed isn't technical, it's business. Implementing a new system that others would need to learn has a cost impact (time taken for people to learn). Have any of your discussions addressed this aspect?

You're seeing the issue in the context of the development of your application, he's assessing it in the context of integration into the business.

Answer 2

My teamleader said that, based on the fact that another piece of the software already uses the table as it is in production, he does not really want to change the structure

This is the current block. Your team lead is weighing up whether it's worth getting that other piece of software changed (and risk assessed, and tested, and deployed), in order to follow your suggestion of breaking up that table.

Since there seems to be a one-to-one link between those fields, I'd also question whether there's a real need to split that one table into three.

Answer 3

In my previous job I was working on a module which was first made years ago at the request of a new client. The new client was migrating to us, and was using an old database, with a badly thought-out structure. Our technical team at the time decided to wholly adopt the structure as it was, in the interest of time.

Eventually everyone forgot about the client and our new product flourished, with new features and interest from other customers. But the bad structure remained, haunting me from the day I joined to the day I left.

Your point of argument is that this should be treated as any other product, and that you should base your design decisions on what will also work in the future. A well thought-out schema will save you countless hours of head-scratching and hacking things together with duct-tape and prayers. I speak from experience here.

The main argument is that it'll save you time and headaches in the future. You should be able to design a half-decent schema and then expose an interface compatible with the client's structure (ie. database view). This gives you all the flexibility you need without locking you in with someone else's bad decisions.

Answer 4

The rules of normalisation are known to him and he know that this structure is breaking them, but that is not enought reasoning for him.

And rightly so. While normalization is often a good thing, there may be good reasons to deviate from it. For quite some time at my current job, during the interview process for new developers we would ask about normalization, and once we had established the candidate knew about normalization, we'd ask whether (s)he could think of reasons why you want to denormalize data.

I already tried to explain to him that it is easier for me to work with a normalized structure, but I failed to do so somehow.

That's an argument, but not a very strong one. Easier to work with is what you pick if the alternatives are equally good in all other aspects.

If you think that normalized data is the right thing for this project, you have to show why it is the right thing. There are benefits and drawbacks about normalization; show your lead why the benefits outweight the drawbacks for this project. That is, you have to show have the general benefits of using normalized data apply to this particular project (and the same for the drawbacks). Take into account that your leads proof-of-concept is piggybacking on an existing datastructure; not being able to that with your suggestion is a drawback (specially if the piggy-backing comes with code reuse).

Answer 5

I've been a team lead before and I know the rules of normalization very well, so I'll interject here.

I've already tried to explain to him that it is easier for me to work with a normalized structure, but somehow I failed to convince him.

Academics of normalization

When you say normalized, exactly which normalization level / standard is right? 2nd normal form? 3rd normal form? Boyce–Codd normal form? 6th normal form?

There are a lot of levels of normalization, each with increasing complexity of rules: https://en.wikipedia.org/wiki/Database_normalization

So by saying that X is better than Y because normalization is good, you are implying a single, unambiguous rule of what is right.

Analysis of normalization and tradeoffs

We normalize primarily to optimize write performance and guarantee data consistency.

Look at this page about GitHub database performance: https://johnnunemaker.com/database-performance-simplified/

Notice how the "how to optimize writes" recommendations are the inverse of the "how to optimize reads" recommendations? This is a general pattern you will find. There exists a tradeoff between read performance and write performance.

The specific normalization condition you are describing is to split a table into 3 tables. Let's assume it's not 1:1 and so some of the tables have a smaller row count and you've reduced the storage footprint. That means that updates to that data set may be more efficient. Now how do you read the data set? By joining 3 tables. Joining requires looking at 3 tables instead of 1, so it will be more expensive.

You've optimized writes at the expense of reads.

In a past role, we found that almost every important query joined in a single particular table just to find a single value. 10% of the cost of most queries was joining to this one table. It was not being used to filter. So duplicating that column (which would never change) onto a few frequently queried tables improved performance by 10% across the board. It was a thoughtful, deliberate decision done by people who knew what they were doing.

Back to your example. If you split that table into 3 tables with a 1:1 relationship, then you always have at least the same number of rows to update, so there is no benefit in terms of writes, but now reads have to join 3 tables. So you're taking a performance cost without a benefit.

Conclusion

Some data is written far more often than it is read. Some data is read far more often than it is written. Since we have a tradeoff between read efficiency and write efficiency, we should optimize for the situation that matches our usage patterns.

Given that you're new to the company, your team lead probably knows more about how this data is used than you do.

Next time

So: How can I convince my teamleader that another data structure would be better?

The fact that you didn't bring this to stackoverflow or the DBA Stack Exchange and ended your question with asking how to convince someone you're right shows that you haven't considered your team lead may actually be right.

Next time, you should approach a disagreement by asking him to explain his concerns and then if you still don't agree, go to stackoverflow.com or dba.stackexchange.com and ask someone to explain the technical merits to you. You should try to see disagreements as a learning opportunity rather than opportunities to force your will on others. Especially when you're new and surrounded by people more knowledgable than yourself.

Answer 6

This appears to me like it's a little bit too technical for this SE. However, let me give you my two cents.

I'm taking a guess here, but from your picture it looks like you've not been working for a long time. You're probably still fresh and enthusiastic about the things you've learned recently. That is commendable.

But it's not always the best idea to go and challenge the status quo when you come into a new place. Especially if you're not already a very senior person that was hired for their expertise (again, I'm guessing).

There might be numerous reasons for this database table to be designed the way it is. They could include, but might not be limited to:

• "historical reasons" (as in someone didn't know any better)
• data needs to be redundant for auditing reasons (like e-com orders keeping the price of each item, and the VAT, instead of looking that up, because it needs to be immutable)
• speed
• too much data in it, refactor and changing the table is going to take forever
• it's "too clever for our developers"

You will find that often in real life, best practices are not always followed. Sometimes that's for business reasons, sometimes it's because there is not enough time and the business values roll-out speed over quality and is either too far away from realising that this will build technical dept or is already so used to technical debt and the cost of maintenance that they don't change, or maybe the person in charge is simply incompetent.

Whether it's business or technical reasons, there are cases where this decision is taken on purpose, after weighing all alternatives. What looks like "the wrong choice" from the outside, might very well be the best choice for this business. From the outside that is very hard to see, especially when you lack experience. It's all too easy to come to conclusions.

When you write new code, you can try to do things the "right" way. Be prepared to defend that, to write documentation for it because the other developers are not as well-trained as you are, or to fight for your improvements because people don't like change. But keep in mind that others' arguments might ultimately be better because the non-academic approach suits the business better.

In the meantime, I suggest to take it slow, learn how the place functions, and do things the way they are done there. Move on to better practices when you understand why they do it the way they do, so you can bring proper arguments. Take away some learnings. There is knowledge to be gained in all of this.

If you absolutely must have this in a normalised way, write an abstraction layer. Show them how your code is better/faster/easier to maintain/prettier/more complicated/whatever and explain why that is preferable, and then convince them to use the abstraction layer as a temporary solution to refactor the existing code, so that you can then refactor the database.

Answer 7

Get evidence.

Weigh up the pro's and con's of each and tell him why your way is better than the current structure and how it would benefit your team/company. Focus on the reasons why your way is better and the reasoning on why you've done it.

Answer 8

Most of the answers here refer to different aspects of the same topic here: This is about software architecture, although on a relatively low level in this case.

Still, there are some very good approaches in software architecture to come to decisions (after all, this is what software architecture is about) which might help you here.

First, collect relevant requirements and constraints. Usually, these should include requirements from a standard like IEC 9126. For example, the maintainability may be increased when the tables are normalized. Also, how do the tables behave over time? If lots of data will be added, one solution may be superior to the other - but this is only relevant if performance is important.

But the existing software already working on the table is a constraint which has to be obeyed. As others suggested, this may require a compatibility layer (e.g. a view). You must include this in your solution design as well.

In general, if you want to change an existing solution like this, prepare your arguments well, with the help of the standard mentioned above (or other standards). If there already exists a utility tree for the software, it should make it much easier for you to collect good arguments (or to detect that the single table maybe is the most feasible solution here).

But just saying, "this data must be normalized because this is how you do it" is indeed not enough in software engineering. There are many technical solutions out there which may look weird to fresh graduates, but when checking all of the constraints and (non-functional) requirements, are totally appropriate. (And of course, there are many more which are not.)

Answer 9

Funnily, I have been in the situation where I had to design, or to decide between, or switch existing tables between both of your variants (in both directions, too!) for keeping history in a RDBMS, so let me play the advocatus diavoli here.

Let me keep it short and simple: yes, normalization is good, and if in doubt normalize. But there are forms of data storage (like this type of historization) which use voluntary, deliberate redundancy for reasons that are not absolutely stupid (for example, performance or ease of use). If this were stackoverflow.SE instead of workplace.SE, we could go into depth, but let's just put it out here.

Another aspect is that sometimes the aspect of doing the (n+1)st iteration of something just like the previous n iterations is a valuable thing in IT, and can dwarf having a new design for every bit you come across. If the new design really is better than the old, it may be worthwhile to hold on to the old until you are able to replace all occurences of the older one at once.

This may all be too technical for workplace.SE, but the point is that your teamleader has a point. You need more than "all tables must be normalized at all times". You have to weigh the history, the future, soft aspects like a similar approach for similar cases, and even, yes, the gut feelings, opinions and biases of the other developers/stakeholders.

My feeling is that unless you can prove with really tight code that your approach is by leaps and bounds better, a colleague may well still find some aspect surrounding the issue that makes it at least arguable whether to stay or go with it, so ... pick your fight wisely. Of course, try to push a bit, but don't lose sleep over it.

Answer 10

To expand a bit on @JozefWoods' excellent answer, it only makes business sense to repay technical debt if the overhead of that debt is likely to outpace the cost of fixing it. That cost includes not just the implementation but any manual testing, documentation, changing every piece of software which interfaces with it, migration, downtime, the risk of introducing bugs and, crucially, opportunity.

A common rule of thumb is to refactor only those parts which you touch when changing the application to implement something else, and then being very careful that the refactoring part of the work is a small part of the total. Doing this consistently, hacking at the technical debt one small piece at a time, can do wonders even in big balls of mud.

Answer 11

The answer here is very short:

Create a demo showing how your normalized structure solves problems for the business.

You mentioned there are other projects using the flat table. The demo should include those other projects as well as new projects.

Normalized tables aren't the best option in every situation. If you want to convince a tech lead to do something, show that lead how your proposal makes everyone's job easier or solves a bigger problem with a working demo - not with text book theory. A demo is a much stronger proposal than any argument alone.

Answer 12

This is a situation where code speaks louder than words. Write one version your way, one version using their way, and one version using your way in the top layer translating to their way in the bottom layer. If your data structure really is superior, you should be able to write code to reflect that. You don't have to spend a ton of time implementing the entire feature 3 ways, just enough to make the point.

Sometimes, when I do this exercise, I find out the way I disliked actually isn't too bad. Usually, I learn things from writing one version that help me write another more cleanly. I nearly always learn how to articulate the pros and cons of both approaches more precisely.

Answer 13

Changing the structure is a bit of a red herring situation to me. The principal question is why the programmers should to cater to a particular implementation in the first place.

The functionality that needs to be covered has to be made available in a good API that provides functions for doing what is required and maps them to some efficient implementation without the details of the implementation leaking out all over the API. Once this is achieved, life becomes easier for the users of the data structure, and swapping out the data structure for something better matching the performance profile becomes easier for the maintainers of the API and underlying structure.

So I'd split your desire into four separate tasks: designing a good neutral API with the required expressivity to support the most efficient way of operating the current and your desired data structure, implementing that API, establishing that API in current code (a usability/maintainability issue), and then you have the situation where proposing a different implementation and checking its performance implications is comparatively cheap.

In theory. In practice this is where you find out that "establishing that API" was not actually complete since people bypassed your API for their favorite hack which breaks by changing the implementation underneath. Or did not actually change anything at all. But then you are in a better position to get that change under wraps, and assuming that your design of the API was good and complete, people will be happy about it in due time. And they better should, because if they aren't, the number of times you can pull that stunt will be limited, so you better get everything right.

Answer 14

This is a great question, and I believe this comes up a lot! First thing to understand is although you are doing a technical job, in a technical field, decisions are rarely made on the basis of technical benefits alone. This is extremely different as what somebody experiences from day a traditional channel of education like university.

My experience has been once something is deployed, business decision making processes tend to overrule technical ones. So you would need to argue from a business perspective instead of referring to design principals such as advocating for a normalized structure to convince the business to change this data schema.

This would mean articulating why changing the data structure will result in a better outcome for the business. You cannot argue that by virtue of your proposed changes adhering better to Normalization that is sufficient to change peoples' opinions.

I suggest you also need to scrutinise the invitation to state your case. First you should try and do some non-invasive method of finding out how willing the other engineers would be to support the decision. Picking when and what battles to fight is important. If there is no appetite for change it wouldn't be productive to proceed, often people are less facilitating of change than they will admit. It might better serve your goal if you acknowledge the reasons why the schema hasn't been changed and concede it shouldn't be changed at this point in time.

However if you wait for the right time and piece of work to fall on your lap that may be a more opportunistic time to try and push your suggestions through then.