1

I created software that imports data and creates records (sales orders) automatically. Records are created in two different ways - A and B - depending on the configuration setting X. My client C is expected to create records in mode A and the setting X was set accordingly. But my colleague delivered his own software upgrade that unintentionally changed the setting X and some batch of records in mode B were created.

Now I have to work on transforming records from format B to format A. It is quite complex work. I need to create one-time code that will be almost useless in the future. My company is burning money in this effort but what else I or our company or our client can do?

One can imagine that our client can import data repeatedly - with the right setting X on and created records in format A and the erroneous records (in format B) one could delete. But that is not possible. The error has been discovered some weeks after the process completed and the process depends on the time and it have impacts on other records too. (cost prices of goods, specifically - they have changed between then and now). So, that is the reason why I am creating specific transformation process to correct the situation.

I wonder - how I or my company or my colleague or our client should handle such situations? It is not the first time when we are creating specific correcting procedures when it appeared that some of our original procedures where incorrect, or partially correct or executed incorrectly at the incorrect values of the setting.

Yes, we devote a lot of time to testing, but still - the error can be? Can't they? How should I handle such situations according to the good practice in the industry? I expect that there is good practice how ERP vendors deliver data corrections when their software failed to work correctly? I hope that I can use such guidance in my situation as well. Currently much of my time involves creating such one-time correcting procedures and it does not create any monetary value at all. I am feeling not worth for my salary.

3
  • Yes, I am doing exactly that - fixing and moving away. I just had a moment of low mood and tried to find some meaning and general guidance. Things happen.
    – TomR
    Apr 6, 2022 at 10:21
  • Your company might want to ask its lawyers. And then your management will tell you what to do. If they tell you to write code that fixes these records then you do it, otherwise you don't. You'll get paid for your work. Whether your management's decision is a good one or not is up to them.
    – gnasher729
    Apr 6, 2022 at 10:22
  • You ask your colleague to come up with the solution to his error.
    – Solar Mike
    Apr 6, 2022 at 13:30

6 Answers 6

6

Yes, we devote a lot of time to testing,

But now you have another thing to test for. It may mean that you need to test with a broader set of data not just the data related to the item being changed. It may mean that the configuration file isn't being correctly tracked in the version control system.

You may have to do more on site testing when a new module is delivered to make sure that there aren't unintended changes. You may need rollback procedures.

Currently much of my time involves creating such one-time correcting procedures and it does not create any monetary value at all.

You are re-creating value. The company broke the customers data, you are fixing it. Management should be worried that this happens too often, because these types of things make customers angry, and they go to other vendors.

We tell people that testing early and often catches bugs when they are cheaper to fix. That guideline may also extend to software deliveries. Bugs that destroy data can be hard to fix. They cost time and money, and usually take place in full view of the customer.

Having been in a similar situation before, being asked to cleanup messes gets old quick. To get out of this cycle your company needs to find away to either minimize the occurrences or to catch the issues faster.

1
  • +1000. Deployment (and realizing installation and upgrading are separate tasks) is ABSOLUTELY something that needs tests. Apr 6, 2022 at 19:33
2

In every organisation, people make mistakes. The question is how to do you mitigate the risk and impact of these mistakes.

In this particular situation, the solution is simple: don't have the generated reports as the "primary record" of the data.

The data should be stored in such a way that the reports are merely a "view" on the underlying data. Which means a minor tweak to the configuration (or even a major one) is fine. The reports can simply get regenerated.

This also protects against bugs that exist when generating the reports. If the bug is found, affected reports can simply be regenerated.

You should note that even if you can write a script that generates report format B from report format A, that may not always be possible. If report format A does not have some field that report format B has, then you have no way of recovering that data.

1

How should I handle such situations according to the good practice in the industry?

You document everything and then follow directions from your manager. If there is any problem involving a client normal practice is for it to be escalated to a higher level for resolution.

1

The problem that needs to be fixed is that a change to the code had such a large negative impact on a client. Fix the problem for your client as efficiently as you can, then have a postmortem to figure out how you can prevent similar problems in the future.

Some things the team may want to discuss:

  • Is there a better way to ensure clients are running with the correct configuration?
  • Should the test suite include client specific tests?
  • How did the change get through code review? Were the "right" people involved in the review?
  • Should there have been better documentation of the setting?

Mistakes happen, so this isn't about blaming someone. It's about the whole team learning from a mistake and getting better. If you improve your process after every mistake, it should reduce how often you have to spend time correcting a mistake in the future.

I suspect that is not the answer you were hoping for, but getting better at fixing costly mistakes is less effective than getting better at not making costly mistakes. Your team will make mistakes; part of improving your process is catching those mistakes before the client is impacted by them, and making it less likely that a mistake will have a large, expensive to fix impact.

0

If there is a program with a software solution, it falls to one of 3 categories:

  1. Requirements & unit test conditions from the customer were insufficient to meet the need. The customer pays for rework.

  2. The specifications and unit test conditions were fine, but the code was bad and wasn't unit tested properly. Your company pays for the re-work.

  3. Units tests and initial code were fine but regression testing uncovers a problem with something else in the system. The customer will usually pay for this if the customer is the expert, your company does if they are the expert, or the costs are split if it's in between.

-1

There is another potential solution. If you know the algorithm to "fix" the records, you can deploy that algorithm in a database "view" in such a way that it only operates on the errant records when they are queried. This way you don't have to update potentially millions of records in the database; you just leave them. They get fixed at presentation time.

P.S. I have a friend that got hit PhD in Computer Science by devising this technique.

2
  • In normal use, and depending on things like the RDBMS being used and data volumes, this is likely to be very inefficient: you have to develop the correction and then it's run with every query rather than just being applied once to the source data. Apr 6, 2022 at 14:52
  • But if you have hundreds of billions of records, it is easier to fix the ones you need to touch that attempt the impossible task of trying to fix all of them even though you will likely never touch 99.999% of them. That is the situation I commonly find myself in.
    – djhallx
    Apr 6, 2022 at 17:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .