Was I too conservative in asking my team to test a production database mass-update script? [closed]

Question

Before you say this is a no-brainer, I should qualify that the script is VERY simple. I agree with the developer who wrote the script that the probability of something going wrong was somewhat remote.

Essentially, the script relabeled a set of historical records (about a half million of them) temporarily to alleviate a critical performance problem for one of our clients. This happened around 4:30pm on a Friday and I think everybody was tired and wanted to sign off for the week.

Our production environment is probably medium to small sized. It hosts hundreds of tenants, averaging a dozen users each, and receives somewhere between 6 and 30 transactions a second. However, we are SOC compliant and need to maintain those credentials to keep some of our larger clients.

I'll also say that I came in to the conversation somewhat heavy handed - I said that "We would" test this script, that there was no question, but I met with such anger from one developer that I decided to roll back that decision and asked the team to vote on it.

The team voted 3 to 2 to test and because I've worked with these people for years, I'm pretty sure they didn't feel compelled to do so. I could always be wrong of course.

I've had VERY BAD experiences in the past with production mass update scripts, and so I'm worried my past caused me to be too conservative.

As the director of the group, I'm always concerned that pushing decisions through with my authority (instead of reasoned arguments) could shut down lines of communication, but this felt like an obvious case to me where testing was required, and I didn't understand why I would need to explain.

After testing for an hour with no problems, we were able to run the production update successfully. The developer who was opposed to testing made several comments during this hour that this work was unnecessary and that there was no chance of anything going wrong. After we finished, the same developer commented that we could have been "in the same place an hour ago."

Was I out of line and too conservative to require the hour of testing?

Answer 1

I will assume a few things here: you have been given the job of Director, not because of some random draw of your name from a hat or you being the nephew of the owner, but rather through the normal interview process and picking an experienced person for the job.

I've had VERY BAD experiences in the past with production mass update scripts, and so I'm worried my past caused me to be too conservative.

Yes. But isn't that what you were hired for? Your experiences? Isn't that why they did hire you, and not a random person of the street?

If your experience says this needs to be tested, and you are the director, then it needs to be tested. Period.

While I agree on your assessment on whether it needs testing or not, that is not the point. Even if I disagreed. You were not hired to ignore you past experiences, you were hired because of them.

Answer 2

Was I out of line and too conservative to require the hour of testing?

Without knowing the details, and the ramifications of a script failure, there's no way to know if you were too conservative or not.

But you are a Director? And you asked the team to vote on it? I don't understand the point of that.

You don't shed responsibility by holding a vote.

If you aren't technically able to make the decision by yourself, you should ask for opinions from those who are technically competent, then make the decision for yourself.

Either way, the buck stops with you, so if some of the team don't like it, that's just unfortunate.

Answer 3

I've had VERY BAD experiences in the past with production mass update scripts, and so I'm worried my past caused me to be too conservative.

That seems to me like your work has too many systematic failures.

You were right to be conservative for something that's so revenue-critical

I'll also say that I came in to the conversation somewhat heavy handed

• I said that "We would" test this script, that there was no question, but I met with such anger from one developer that I decided to roll back that decision and asked the team to vote on it.

The team voted 3 to 2 to test and because I've worked with these people for years, I'm pretty sure they didn't feel compelled to do so. I could always be wrong of course.

Work is not a democracy. If the vote had gone the other way and if things had gone wrong, you would have been on the hook for the blame.

With that said, deploying such a script at 4:30PM on a Friday is asking for trouble. Everyone is likely to be in a rush and make mistakes. And if something does go wrong, that means someone is going to be stuck working on Friday night or during the weekend.

Ask yourself. Did that angry developer usually get off at 4:30PM or on 5PM on Fridays? Was someone waiting for him? And how would you like it if a boss waited until the last minute before you're just about to clock off before they asked you to do something really important?

Could this kind of script have been run earlier in the week? Or later? May be next Monday, you should run a post-mortem debriefing session. If this was really a surprise that such a script needed be run at the very last minute on a Friday, maybe you need to change your systems to give yourself more leeway the next time around.

After we finished, the same developer commented that we could have been "in the same place an hour ago."

And yes, that's what work is. Sometimes, a lot of work is incredibly redundant. Surgeons, accountants, pilots, programmers, DBAs, etc. Most of the work professionals do is making sure they're not making mistakes.

Answer 4

The Workplace issue here is that nothing critical should have been scheduled for 4:30 PM on a Friday unless waiting for Monday would be a disaster.

Nothing should ever be allowed to run up to a hard deadline; you should be planning to close things out with buffer time, and be planning to defer late changes to the next release -- unless they are absolutely critical AND discovered late AND you absolutely can't slip the deadline (megabucks at stake), in which case you tell your staff exactly that and they will grumble but accept it.

If the problem is that you didn't realize you needed to plan for testing, or didn't realize the team was running late, or something of that sort ... that's something you need to work on fixing as director. The fact that you were scrambling is a symptom; address the disease.

Answer 5

I've been taught some important rules along my life, and two of them could apply here : 1. never ship on friday 2. measure twice as you can only saw once.

The latter means that I never ever did something wrong when testing first. I invested some little time, but never wasted it. But made big mistakes sometimes when I wanted to do anything without proper testing. They know that, even if they just want to go home or wait for monday morning to do this duty.

The former, in my opinion, would have needed some communication, if you really had to release it before the week-end. You can't expect people to welcome your decision when they're 30 minutes shy of a trip back home (or of any personal scheduled stuff).

When I need to ask this extra step to people, I first show empathy, because I really feel it. You know you're asking for something almost nobody will enjoy, no matter how professional and/or paid they are. So, be nice. It's just a small step for you, but a more important one for them.

Let them understand that you know what they feel and that you're asking for a last drop of courage. Motivate them so that this last step is done quickly and safely, then thank them for the good job.

People will stick to your ideas if they see that it has to be done, and if it can be done as a (last) team effort. Not if they're forced to do it. I'm not sure that your vote was the best thing to do then. You also opened the door to unnecessary and bitter comments afterwards.

Answer 6

I think the most likely cause of the conflict is a perceived contradiction between:

1. justifying the testing process as an act of responsibility and caution; and

2. choosing to suddenly embark on the process at 4.30pm on a Friday afternoon, which is irresponsible to the extent that people are already tired, haven't had time to plan the work, and will be in a poor position to correct any mistakes they make in the process.

Insofar as the circumstances may have been an emergency, there may also be conflict about whether it is as unusual as a building fire and deserves exceptional attention, or whether it happens frequently enough or is minor enough that it could have been left until Monday morning (or should be covered by a dedicated person who is paid to be happy to routinely work later, or to be on-call).

The test of whether it is a true emergency is whether, had it happened at 1am instead of 4.30pm, would you be ringing telephones and knocking on doors to get all hands in attendance, or would you deal with it later in normal working hours?

So the OP casts the question as whether his choice to test was being too conservative or not. The real issue may be whether he'd already been too cavalier by choosing to do that work at all, rather than schedule it at a more appropriate time.

Answer 7

TBH I think this is more appropriate on say the DBA SE or similar - but here's my thoughts:

If a SQL operation is making a change (Upsert or delete) on multiple records or in Bulk - then it should be tested.

Now, sometimes there's a little lee-way, for example if I was doing a:

update Table

set colA = "Foo"

where colA = "Bar"

Yes, it's an update operation, but since the rollback is as simple as running it in reverse (assuming there are no other values in Col A that equal Bar, then for something like that, I might say 'It's low risk enough that we can forgo testing'.

That said, since it took an Hour of testing, I'm presuming that it wasn't such a simple change as the above.

I often have a similar healthy tension with my Boss, I'm a little more gung-ho, he's a little more cautious and together what we do works.

In terms of your specific situation, given that it took an hour to test, I don't think you were being over cautious, especially on No-Change Friday.

But let me outline some factors and ideas:

1: What was the Dollar Value lost to your client for the extra hour of downtime?
2: What would be the Dollar Value lost to your client if you had to do a full Database restore and then have to fix the problem again, over the weekend, paying OT?
3: How quickly could you roll back the change if something unintended happened? If the roll-back was a 5 minute process, then an hour of testing seems overkill, whereas if roll-back was a 6 hour task then...
4: If the Script fails (either to fix the underlying problem or creates additional issues) - what would be the time required to fix those additional issues?

It's not an easy tightrope to walk, but as a Manager - the above are what should guide your decision. Sometimes the cost to test is greater than the cost to rollback, in which case leniency and expediency is fine. In most cases though, the cost roll back is greater than the cost to test, hence we test.

As the old Adage says - Measure twice, Cut once.