32

I am a new hire and my current task is to implement an editor for some data. My teamleader had already made a proof-of-concept of this editor and laid out the data structure for it.

Currently the data structure is a one-to-many relation (with version history of old entries and draft states for new entries) all in one table.
I would like to change that to a normalized version spanning three tables. (You can see the detailed "pseudo"-structure in the edit history if interested. I cut it out to focus more on a non-technical side of my question.)

My teamleader said that, based on the fact that another piece of the software already uses the table as-is in production, he does not really want to change the structure. But he also said that if I bring him valid reason(s) for the change, we will change it.

He knows the rules of normalisation and he knows that this structure breaks them, but it is not enough for him. I've already tried to explain to him that it is easier for me to work with a normalized structure, but somehow I failed to convince him. I think he can't get my point because he does not work on this part anymore.

So: How can I convince my teamleader that another data structure would be better?

closed as off-topic by gnat, Fattie, Dukeling, 520, Magisch Feb 12 at 17:10

  • This question does not appear to be about the workplace within the scope defined in the help center.
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    Comments are not for extended discussion; this conversation has been moved to chat. – Jane S Feb 12 at 5:15
  • 1
    Welcome to The Workplace. It might be best to look at why your team leader wants to avoid this change. Would it break a business critical feature? Do other tools query the database directly? Is there a contract with a customer which stipulates it will be that way? (Also, as an aside, could you build a database view from the normalised data with would provide the rows which are currently in the all-in-one table?) – AJFaraday Feb 12 at 9:31
  • Why not ask this on stack overflow? A database structure is hidden by the "data access layer", so it's invisible to the customer and of little impact to developers, given that the "DAO" works. Rewriting a database in production takes a lot of effort to implement, test and migrate. Maybe you should follow your team leaders advice, and not try to refactor or break up things right away, especially when youre a new hire? Add it to the backlog ("TODO" list) – vikingsteve Feb 12 at 11:02
  • I checked your structures, and.. I have to say, I would team up with your team leader. Normalization is nice but it seems it would be a hinder for this use-case. Keep in mind that it is a guideline, not a rule. A GOOD guideline, mind you, but it still can be ignored if the problem asks for a different strategy. – T. Sar Feb 12 at 13:04
  • So basically you want to change an operational solution that has been tested to work by another (untested and undeveloped) one just on theoretical grounds? His suggestion that you come up with valid reasons to do so (valid as in operationally valid) is a very reasonable, nice and polite one. Especially when, after pure theory, your second to best reason is that "it is easier for you". It may be easier for you, but is it for the company workflow, operation, clients, etc? – busman Feb 12 at 16:04

14 Answers 14

147

It's worth looking at this from the other perspective, namely, that of the team lead:

You have two different structures, one is tested a proven, presumably with an existing number of people supporting similar-structured applications. The second has been brought to you, with technical reasoning, but, crucially, without business reasoning.

Rather than not getting your point because he doesn't work in the area any more, it's possible that you're not getting his. The reasoning needed isn't technical, it's business. Implementing a new system that others would need to learn has a cost impact (time taken for people to learn). Have any of your discussions addressed this aspect?

You're seeing the issue in the context of the development of your application, he's assessing it in the context of integration into the business.

  • 51
    Also, its even in the context of development not always useful to normalize a datastructure. There are pros and cons as usual and it depends on the application itself which you weight more – XtremeBaumer Feb 11 at 13:02
  • 3
    @XtremeBaumer this is true. For example I like to consider which structure would be easy/efficient/simple to hydrate with the ORM in use (if any). Fully normalized structures become difficult to map sometimes. It sure as hell would be easier to map 1 flat table to a list and sort out the various states in the domain code. – Reasurria Feb 11 at 13:06
  • 3
    And once you have enough joining tables setup, it becomes an absolute nightmare to trace where the data is, and then also sifting through all the various different foreign keys because different people happen to have found different ways to create the relation... If you work on Financial systems, you'll know what I mean... – Nelson Feb 12 at 3:39
  • I think it's not just an issue of business value but also of cost in relation to the business value. Often times, the cost involved in eliminating tech debt outweighs the benefits so it's very hard to convince management to approve that work. You need to demonstrate that investing this time and effort now will save at least as much time and effort in future maintenance costs. – aleppke Feb 12 at 22:48
  • Thanks for the heads-up, I stopped trying to push to normalize the data. I implemented an abstraction layer like @simbabque suggested in his answer. I already talked to my lead about this and he was very interestend and said he want to have a look on it, when I am done. Best of both worlds. | Thanks to all for the input you all gave me. – Mischa Feb 14 at 11:30
39

My teamleader said that, based on the fact that another piece of the software already uses the table as it is in production, he does not really want to change the structure

This is the current block. Your team lead is weighing up whether it's worth getting that other piece of software changed (and risk assessed, and tested, and deployed), in order to follow your suggestion of breaking up that table.

Since there seems to be a one-to-one link between those fields, I'd also question whether there's a real need to split that one table into three.

  • It isn't really a one-to-one... It is a one-to-many. Everytime the data gets changed in the editor a new entry is added to the table the old entry is set to "inactive" and the new entry (with the same "group" and "otherGroup") but status "active" will be inserted to the table... Also there is a "draft" status. which i need to consider when editing but not when displaying... – Mischa Feb 11 at 10:18
  • 8
    Would it really be better having inactive and draft in separate tables? Now the table structures need to be kept in sync, and rows need to be deleted/created purely for the sake of changing their status. It could be better / worse; but from a workplace point of view you need to consider the idea that your data structure might not actually be better. – gamesmad Feb 11 at 14:24
  • 4
    @Mischa TBH, although we don't have all the information, from what you've just told us the current layout sounds sensible. At the very least, if it isn't, you'll need to work on your persuasion speech. :) – Lightness Races in Orbit Feb 11 at 20:07
  • 3
    @Mischa the behavior you are explaining is a manual version of SQL Server's new "history" feature, where updates are converted by SQL Server on the fly into new rows with the old one archived but queriable with a specific query. And the denormalisation you are mentioning could be quite desirable, as it locks in a snap-shot of the state of the fields at the time of the edit. – Graham Feb 11 at 21:34
  • 1
    @Graham Mind providing a link to "SQL Server's new 'history' feature" – Nightwolf Feb 12 at 6:57
10

In my previous job I was working on a module which was first made years ago at the request of a new client. The new client was migrating to us, and was using an old database, with a badly thought-out structure. Our technical team at the time decided to wholly adopt the structure as it was, in the interest of time.

Eventually everyone forgot about the client and our new product flourished, with new features and interest from other customers. But the bad structure remained, haunting me from the day I joined to the day I left.

Your point of argument is that this should be treated as any other product, and that you should base your design decisions on what will also work in the future. A well thought-out schema will save you countless hours of head-scratching and hacking things together with duct-tape and prayers. I speak from experience here.

The main argument is that it'll save you time and headaches in the future. You should be able to design a half-decent schema and then expose an interface compatible with the client's structure (ie. database view). This gives you all the flexibility you need without locking you in with someone else's bad decisions.

  • 6
    A manager wants figures. A change that may save money in a somewhat distant (or not so distant, it doesn't actually matters) future, will cost money now if the change is cascading on other modules. When dealing with management (even if technically savy), a vague possibility of undefined improvement stands no chance against a granted expense... Provide numbers or you get no traction... – Paolo Feb 11 at 20:18
  • 2
    @Paolo I can't disagree with that. – rath Feb 11 at 20:47
10

The rules of normalisation are known to him and he know that this structure is breaking them, but that is not enought reasoning for him.

And rightly so. While normalization is often a good thing, there may be good reasons to deviate from it. For quite some time at my current job, during the interview process for new developers we would ask about normalization, and once we had established the candidate knew about normalization, we'd ask whether (s)he could think of reasons why you want to denormalize data.

I already tried to explain to him that it is easier for me to work with a normalized structure, but I failed to do so somehow.

That's an argument, but not a very strong one. Easier to work with is what you pick if the alternatives are equally good in all other aspects.

If you think that normalized data is the right thing for this project, you have to show why it is the right thing. There are benefits and drawbacks about normalization; show your lead why the benefits outweight the drawbacks for this project. That is, you have to show have the general benefits of using normalized data apply to this particular project (and the same for the drawbacks). Take into account that your leads proof-of-concept is piggybacking on an existing datastructure; not being able to that with your suggestion is a drawback (specially if the piggy-backing comes with code reuse).

9

I've been a team lead before and I know the rules of normalization very well, so I'll interject here.

I've already tried to explain to him that it is easier for me to work with a normalized structure, but somehow I failed to convince him.

Academics of normalization

When you say normalized, exactly which normalization level / standard is right? 2nd normal form? 3rd normal form? Boyce–Codd normal form? 6th normal form?

There are a lot of levels of normalization, each with increasing complexity of rules: https://en.wikipedia.org/wiki/Database_normalization

So by saying that X is better than Y because normalization is good, you are implying a single, unambiguous rule of what is right.

Analysis of normalization and tradeoffs

We normalize primarily to optimize write performance and guarantee data consistency.

Look at this page about GitHub database performance: https://johnnunemaker.com/database-performance-simplified/

Notice how the "how to optimize writes" recommendations are the inverse of the "how to optimize reads" recommendations? This is a general pattern you will find. There exists a tradeoff between read performance and write performance.

The specific normalization condition you are describing is to split a table into 3 tables. Let's assume it's not 1:1 and so some of the tables have a smaller row count and you've reduced the storage footprint. That means that updates to that data set may be more efficient. Now how do you read the data set? By joining 3 tables. Joining requires looking at 3 tables instead of 1, so it will be more expensive.

You've optimized writes at the expense of reads.

In a past role, we found that almost every important query joined in a single particular table just to find a single value. 10% of the cost of most queries was joining to this one table. It was not being used to filter. So duplicating that column (which would never change) onto a few frequently queried tables improved performance by 10% across the board. It was a thoughtful, deliberate decision done by people who knew what they were doing.

Back to your example. If you split that table into 3 tables with a 1:1 relationship, then you always have at least the same number of rows to update, so there is no benefit in terms of writes, but now reads have to join 3 tables. So you're taking a performance cost without a benefit.

Conclusion

Some data is written far more often than it is read. Some data is read far more often than it is written. Since we have a tradeoff between read efficiency and write efficiency, we should optimize for the situation that matches our usage patterns.

Given that you're new to the company, your team lead probably knows more about how this data is used than you do.

Next time

So: How can I convince my teamleader that another data structure would be better?

The fact that you didn't bring this to stackoverflow or the DBA Stack Exchange and ended your question with asking how to convince someone you're right shows that you haven't considered your team lead may actually be right.

Next time, you should approach a disagreement by asking him to explain his concerns and then if you still don't agree, go to stackoverflow.com or dba.stackexchange.com and ask someone to explain the technical merits to you. You should try to see disagreements as a learning opportunity rather than opportunities to force your will on others. Especially when you're new and surrounded by people more knowledgable than yourself.

8

This appears to me like it's a little bit too technical for this SE. However, let me give you my two cents.

I'm taking a guess here, but from your picture it looks like you've not been working for a long time. You're probably still fresh and enthusiastic about the things you've learned recently. That is commendable.

But it's not always the best idea to go and challenge the status quo when you come into a new place. Especially if you're not already a very senior person that was hired for their expertise (again, I'm guessing).

There might be numerous reasons for this database table to be designed the way it is. They could include, but might not be limited to:

  • "historical reasons" (as in someone didn't know any better)
  • data needs to be redundant for auditing reasons (like e-com orders keeping the price of each item, and the VAT, instead of looking that up, because it needs to be immutable)
  • speed
  • too much data in it, refactor and changing the table is going to take forever
  • it's "too clever for our developers"

You will find that often in real life, best practices are not always followed. Sometimes that's for business reasons, sometimes it's because there is not enough time and the business values roll-out speed over quality and is either too far away from realising that this will build technical dept or is already so used to technical debt and the cost of maintenance that they don't change, or maybe the person in charge is simply incompetent.

Whether it's business or technical reasons, there are cases where this decision is taken on purpose, after weighing all alternatives. What looks like "the wrong choice" from the outside, might very well be the best choice for this business. From the outside that is very hard to see, especially when you lack experience. It's all too easy to come to conclusions.

When you write new code, you can try to do things the "right" way. Be prepared to defend that, to write documentation for it because the other developers are not as well-trained as you are, or to fight for your improvements because people don't like change. But keep in mind that others' arguments might ultimately be better because the non-academic approach suits the business better.

In the meantime, I suggest to take it slow, learn how the place functions, and do things the way they are done there. Move on to better practices when you understand why they do it the way they do, so you can bring proper arguments. Take away some learnings. There is knowledge to be gained in all of this.

If you absolutely must have this in a normalised way, write an abstraction layer. Show them how your code is better/faster/easier to maintain/prettier/more complicated/whatever and explain why that is preferable, and then convince them to use the abstraction layer as a temporary solution to refactor the existing code, so that you can then refactor the database.

  • 2
    I agree with all of this, but it does suggest that the OP is correct in that his approach is better. You are correct to say best practices are not always followed, but the phrasing sounds like a cop-out. I'd emphasize that best practices are best practices, not rules, and sometimes deviating makes sense. Normalization, in particular, often makes sense to give up in the interest of performance and scale. – Brandon Feb 11 at 22:23
  • @Brandon rereading my answer I agree with you that it's not clear enough. I'll rephrase in the morning. Had intended to make the point you are emphazising, but got carried away a little bit. – simbabque Feb 11 at 23:21
3

Get evidence.

Weigh up the pro's and con's of each and tell him why your way is better than the current structure and how it would benefit your team/company. Focus on the reasons why your way is better and the reasoning on why you've done it.

2

Most of the answers here refer to different aspects of the same topic here: This is about software architecture, although on a relatively low level in this case.

Still, there are some very good approaches in software architecture to come to decisions (after all, this is what software architecture is about) which might help you here.

First, collect relevant requirements and constraints. Usually, these should include requirements from a standard like IEC 9126. For example, the maintainability may be increased when the tables are normalized. Also, how do the tables behave over time? If lots of data will be added, one solution may be superior to the other - but this is only relevant if performance is important.

But the existing software already working on the table is a constraint which has to be obeyed. As others suggested, this may require a compatibility layer (e.g. a view). You must include this in your solution design as well.

In general, if you want to change an existing solution like this, prepare your arguments well, with the help of the standard mentioned above (or other standards). If there already exists a utility tree for the software, it should make it much easier for you to collect good arguments (or to detect that the single table maybe is the most feasible solution here).

But just saying, "this data must be normalized because this is how you do it" is indeed not enough in software engineering. There are many technical solutions out there which may look weird to fresh graduates, but when checking all of the constraints and (non-functional) requirements, are totally appropriate. (And of course, there are many more which are not.)

2

Funnily, I have been in the situation where I had to design, or to decide between, or switch existing tables between both of your variants (in both directions, too!) for keeping history in a RDBMS, so let me play the advocatus diavoli here.

Let me keep it short and simple: yes, normalization is good, and if in doubt normalize. But there are forms of data storage (like this type of historization) which use voluntary, deliberate redundancy for reasons that are not absolutely stupid (for example, performance or ease of use). If this were stackoverflow.SE instead of workplace.SE, we could go into depth, but let's just put it out here.

Another aspect is that sometimes the aspect of doing the (n+1)st iteration of something just like the previous n iterations is a valuable thing in IT, and can dwarf having a new design for every bit you come across. If the new design really is better than the old, it may be worthwhile to hold on to the old until you are able to replace all occurences of the older one at once.

This may all be too technical for workplace.SE, but the point is that your teamleader has a point. You need more than "all tables must be normalized at all times". You have to weigh the history, the future, soft aspects like a similar approach for similar cases, and even, yes, the gut feelings, opinions and biases of the other developers/stakeholders.

My feeling is that unless you can prove with really tight code that your approach is by leaps and bounds better, a colleague may well still find some aspect surrounding the issue that makes it at least arguable whether to stay or go with it, so ... pick your fight wisely. Of course, try to push a bit, but don't lose sleep over it.

1

To expand a bit on @JozefWoods' excellent answer, it only makes business sense to repay technical debt if the overhead of that debt is likely to outpace the cost of fixing it. That cost includes not just the implementation but any manual testing, documentation, changing every piece of software which interfaces with it, migration, downtime, the risk of introducing bugs and, crucially, opportunity.

A common rule of thumb is to refactor only those parts which you touch when changing the application to implement something else, and then being very careful that the refactoring part of the work is a small part of the total. Doing this consistently, hacking at the technical debt one small piece at a time, can do wonders even in big balls of mud.

  • One important facet you omit is the cost of mistakes or incorrect data. Such data integrity concerns are one of major advantages of normalization. (They centralize a piece of data in a single place, ensuring that only one edit is ever required to update it.) – jpmc26 Feb 12 at 6:03
0

The answer here is very short:

Create a demo showing how your normalized structure solves problems for the business.

You mentioned there are other projects using the flat table. The demo should include those other projects as well as new projects.

Normalized tables aren't the best option in every situation. If you want to convince a tech lead to do something, show that lead how your proposal makes everyone's job easier or solves a bigger problem with a working demo - not with text book theory. A demo is a much stronger proposal than any argument alone.

0

This is a situation where code speaks louder than words. Write one version your way, one version using their way, and one version using your way in the top layer translating to their way in the bottom layer. If your data structure really is superior, you should be able to write code to reflect that. You don't have to spend a ton of time implementing the entire feature 3 ways, just enough to make the point.

Sometimes, when I do this exercise, I find out the way I disliked actually isn't too bad. Usually, I learn things from writing one version that help me write another more cleanly. I nearly always learn how to articulate the pros and cons of both approaches more precisely.

0

Changing the structure is a bit of a red herring situation to me. The principal question is why the programmers should to cater to a particular implementation in the first place.

The functionality that needs to be covered has to be made available in a good API that provides functions for doing what is required and maps them to some efficient implementation without the details of the implementation leaking out all over the API. Once this is achieved, life becomes easier for the users of the data structure, and swapping out the data structure for something better matching the performance profile becomes easier for the maintainers of the API and underlying structure.

So I'd split your desire into four separate tasks: designing a good neutral API with the required expressivity to support the most efficient way of operating the current and your desired data structure, implementing that API, establishing that API in current code (a usability/maintainability issue), and then you have the situation where proposing a different implementation and checking its performance implications is comparatively cheap.

In theory. In practice this is where you find out that "establishing that API" was not actually complete since people bypassed your API for their favorite hack which breaks by changing the implementation underneath. Or did not actually change anything at all. But then you are in a better position to get that change under wraps, and assuming that your design of the API was good and complete, people will be happy about it in due time. And they better should, because if they aren't, the number of times you can pull that stunt will be limited, so you better get everything right.

0

This is a great question, and I believe this comes up a lot! First thing to understand is although you are doing a technical job, in a technical field, decisions are rarely made on the basis of technical benefits alone. This is extremely different as what somebody experiences from day a traditional channel of education like university.

My experience has been once something is deployed, business decision making processes tend to overrule technical ones. So you would need to argue from a business perspective instead of referring to design principals such as advocating for a normalized structure to convince the business to change this data schema.

This would mean articulating why changing the data structure will result in a better outcome for the business. You cannot argue that by virtue of your proposed changes adhering better to Normalization that is sufficient to change peoples' opinions.

I suggest you also need to scrutinise the invitation to state your case. First you should try and do some non-invasive method of finding out how willing the other engineers would be to support the decision. Picking when and what battles to fight is important. If there is no appetite for change it wouldn't be productive to proceed, often people are less facilitating of change than they will admit. It might better serve your goal if you acknowledge the reasons why the schema hasn't been changed and concede it shouldn't be changed at this point in time.

However if you wait for the right time and piece of work to fall on your lap that may be a more opportunistic time to try and push your suggestions through then.

Not the answer you're looking for? Browse other questions tagged or ask your own question.