Before you say this is a no-brainer, I should qualify that the script is VERY simple. I agree with the developer who wrote the script that the probability of something going wrong was somewhat remote.

Essentially, the script relabeled a set of historical records (about a half million of them) temporarily to alleviate a critical performance problem for one of our clients. This happened around 4:30pm on a Friday and I think everybody was tired and wanted to sign off for the week.

Our production environment is probably medium to small sized. It hosts hundreds of tenants, averaging a dozen users each, and receives somewhere between 6 and 30 transactions a second. However, we are SOC compliant and need to maintain those credentials to keep some of our larger clients.

I'll also say that I came in to the conversation somewhat heavy handed - I said that "We would" test this script, that there was no question, but I met with such anger from one developer that I decided to roll back that decision and asked the team to vote on it.

The team voted 3 to 2 to test and because I've worked with these people for years, I'm pretty sure they didn't feel compelled to do so. I could always be wrong of course.

I've had VERY BAD experiences in the past with production mass update scripts, and so I'm worried my past caused me to be too conservative.

As the director of the group, I'm always concerned that pushing decisions through with my authority (instead of reasoned arguments) could shut down lines of communication, but this felt like an obvious case to me where testing was required, and I didn't understand why I would need to explain.

After testing for an hour with no problems, we were able to run the production update successfully. The developer who was opposed to testing made several comments during this hour that this work was unnecessary and that there was no chance of anything going wrong. After we finished, the same developer commented that we could have been "in the same place an hour ago."

Was I out of line and too conservative to require the hour of testing?

  • 20
    "there was no chance of anything going wrong." - I can only assume this developer is a rookie. Anybody with more than 5 mins experience in software development knows this is laughably false. Dec 11, 2022 at 1:14
  • 2
    It's impossible for us to answer the question posed and off-topic. Being a good directior involves weighing up a whole bunch of factors, and not deferring to a vote, or asking internet strangers. Dec 11, 2022 at 8:51
  • 3
    "[..] there was no chance of anything going wrong" I have regularly said this (and still do), only to have it sometimes blow up in my face. If you want to take a chance, it can be acceptable, but you do need to know how to recover if it does blow up in your face. Dec 11, 2022 at 12:03
  • 1
    @LaconicDroid Indeed, and a manager should never leave someone with that level of overconfidence near a production database on a Friday afternoon unsupervised! Would love to know if they are pushing db scripts out via some controlled procedure that verifies everything and keeps an audit trail, hence the time lag, or they just let devs fire up SSMS/pgadmin/SqlDeveloper/whatever... Dec 11, 2022 at 12:12
  • 1
    Also the mumbling developer should be lectured on being a team player and respect the team decisions. If he cannot do that the team needs one that can instead. Dec 11, 2022 at 16:22

7 Answers 7


I will assume a few things here: you have been given the job of Director, not because of some random draw of your name from a hat or you being the nephew of the owner, but rather through the normal interview process and picking an experienced person for the job.

I've had VERY BAD experiences in the past with production mass update scripts, and so I'm worried my past caused me to be too conservative.

Yes. But isn't that what you were hired for? Your experiences? Isn't that why they did hire you, and not a random person of the street?

If your experience says this needs to be tested, and you are the director, then it needs to be tested. Period.

While I agree on your assessment on whether it needs testing or not, that is not the point. Even if I disagreed. You were not hired to ignore you past experiences, you were hired because of them.

  • 1
    Agreed. Thanks for bringing some clarity to the issue. Dec 12, 2022 at 14:17
  • I will say that the test could have easily been put off until the next Monday because it wasn't important. If it was important then it shows poor planning on the person who decided that they needed to test it at 4:30 PM.
    – Questor
    Dec 12, 2022 at 23:12
  • 1
    @Questor How do you "plan better" for a critical problem reported for example at 14:00 and fixed on dev at 16:30? I'm not sure what your job is, but where I work, "critical problems" don't get planned in advance or only conveniently happen at 09:30 on Tuesdays.
    – nvoigt
    Dec 13, 2022 at 7:17
  • Good testing prevents critical problems. Having critical problems that appear out of nowhere means that you are bad at testing. Because you didn't discover them before deployment...
    – Questor
    Dec 13, 2022 at 17:35
  • 1
    Indeed. Name a single company that is good enough at testing, that they never have critical errors? We are all just humans and so we will at all times have critical problems pop up. Good companies less, bad companies more. But it will never be zero.
    – nvoigt
    Dec 13, 2022 at 20:12

Was I out of line and too conservative to require the hour of testing?

Without knowing the details, and the ramifications of a script failure, there's no way to know if you were too conservative or not.

But you are a Director? And you asked the team to vote on it? I don't understand the point of that.

You don't shed responsibility by holding a vote.

If you aren't technically able to make the decision by yourself, you should ask for opinions from those who are technically competent, then make the decision for yourself.

Either way, the buck stops with you, so if some of the team don't like it, that's just unfortunate.

  • @joi-strazzere Thanks you for your answer. Would it be fair to summarize your answer as, "You are the director, so if you are technically competent, you should have stuck with requiring the testing regardless of what anybody else thought?" Dec 10, 2022 at 22:33
  • 1
    @DaveWelling You don't even need to be technically competent, you just need to understand the risk factors. You then make a judgement call. Dec 11, 2022 at 8:50
  • 6
    "You don't shed responsibility by holding a vote." - I think the purpose of the vote was not to shed his own responsibilities, but to examine collective opinion (and, as the case may be, add weight to his own judgment). You have to be careful about acting like a dictator amongst other experts; you've either got to enjoy an enormous level of trust and deference (rarely the case), or you're quickly going to be picking up the pieces of a shattered staff function where all the better birds have flown.
    – Steve
    Dec 11, 2022 at 10:15
  • 2
    @JoeStrazzere, you might prefer it not to be a democracy, but in software teams, the opinion of those executing the work often has real clout, departures often lead to an awful sting for the productivity of the function left behind (but equal or higher pay for the leaver in a new role), and managers are often no more senior in age or technical skill than the staff they manage. Think of it like managing a group of surgeons - an administrative manager, even a competent surgeon, can't simply dictate medical judgments to others who themselves have strong expertise on medical matters.
    – Steve
    Dec 11, 2022 at 14:42
  • 3
    @JoeStrazzere, well it is to do with your preferences, because you are urging him towards a particular mentality and behavioural style. Little of what is real can be inferred from job titles, and clearly the overall tenor of his post suggests he works in an environment where he relies on maintaining collegiate relationships, and where a style of issuing arbitrary directions would not wash for long. To reiterate my original point, he is not shedding his responsibilities, he is performing them by ensuring that potential conflicts are managed effectively.
    – Steve
    Dec 11, 2022 at 20:57

I've had VERY BAD experiences in the past with production mass update scripts, and so I'm worried my past caused me to be too conservative.

That seems to me like your work has too many systematic failures.

You were right to be conservative for something that's so revenue-critical

I'll also say that I came in to the conversation somewhat heavy handed

  • I said that "We would" test this script, that there was no question, but I met with such anger from one developer that I decided to roll back that decision and asked the team to vote on it.

The team voted 3 to 2 to test and because I've worked with these people for years, I'm pretty sure they didn't feel compelled to do so. I could always be wrong of course.

Work is not a democracy. If the vote had gone the other way and if things had gone wrong, you would have been on the hook for the blame.

With that said, deploying such a script at 4:30PM on a Friday is asking for trouble. Everyone is likely to be in a rush and make mistakes. And if something does go wrong, that means someone is going to be stuck working on Friday night or during the weekend.

Ask yourself. Did that angry developer usually get off at 4:30PM or on 5PM on Fridays? Was someone waiting for him? And how would you like it if a boss waited until the last minute before you're just about to clock off before they asked you to do something really important?

Could this kind of script have been run earlier in the week? Or later? May be next Monday, you should run a post-mortem debriefing session. If this was really a surprise that such a script needed be run at the very last minute on a Friday, maybe you need to change your systems to give yourself more leeway the next time around.

After we finished, the same developer commented that we could have been "in the same place an hour ago."

And yes, that's what work is. Sometimes, a lot of work is incredibly redundant. Surgeons, accountants, pilots, programmers, DBAs, etc. Most of the work professionals do is making sure they're not making mistakes.

  • "you would have been on the hook for the blame" - it seems quite reasonable to think that they would all have been on the hook for the cleanup to some degree. As you say, the real issue here might be the timing of the initial work and the failure to build in after-hours cover (by means other than sweating those already working 9-5).
    – Steve
    Dec 11, 2022 at 10:26
  • @Steve, It doesn't matter if some of the underlings are on the hook for the blame. When a system is behaving poorly, management gets the blame. The buck stops with management. Dec 11, 2022 at 20:52
  • If that were the case, then he'd only be on the hook if he was the highest manager. Really, different environments have different attitudes to blame. A manager of professional staff would not normally be blamed for misjudgment, if it ensued from acting in accordance with the majority opinion at the time the judgment was made. Even if they would be blamed, that could be incentive for keeping the team on your side, to provide a degree of protection and solidarity against unreasonable superiors - in contrast to your assumption that the manager would prefer to stand alone on their own judgment.
    – Steve
    Dec 11, 2022 at 21:29
  • 1
    @Steve A workplace is not a democracy. If you make a bad call, it really doesn't matter how many people made the same call. The only thing that matters is if you should have known better. Dec 12, 2022 at 7:42
  • @GregoryCurrie, why would he have known better, if several other professionals didn't? That's the point you're not engaging with.
    – Steve
    Dec 12, 2022 at 17:29

The Workplace issue here is that nothing critical should have been scheduled for 4:30 PM on a Friday unless waiting for Monday would be a disaster.

Nothing should ever be allowed to run up to a hard deadline; you should be planning to close things out with buffer time, and be planning to defer late changes to the next release -- unless they are absolutely critical AND discovered late AND you absolutely can't slip the deadline (megabucks at stake), in which case you tell your staff exactly that and they will grumble but accept it.

If the problem is that you didn't realize you needed to plan for testing, or didn't realize the team was running late, or something of that sort ... that's something you need to work on fixing as director. The fact that you were scrambling is a symptom; address the disease.


I've been taught some important rules along my life, and two of them could apply here : 1. never ship on friday 2. measure twice as you can only saw once.

The latter means that I never ever did something wrong when testing first. I invested some little time, but never wasted it. But made big mistakes sometimes when I wanted to do anything without proper testing. They know that, even if they just want to go home or wait for monday morning to do this duty.

The former, in my opinion, would have needed some communication, if you really had to release it before the week-end. You can't expect people to welcome your decision when they're 30 minutes shy of a trip back home (or of any personal scheduled stuff).

When I need to ask this extra step to people, I first show empathy, because I really feel it. You know you're asking for something almost nobody will enjoy, no matter how professional and/or paid they are. So, be nice. It's just a small step for you, but a more important one for them.

Let them understand that you know what they feel and that you're asking for a last drop of courage. Motivate them so that this last step is done quickly and safely, then thank them for the good job.

People will stick to your ideas if they see that it has to be done, and if it can be done as a (last) team effort. Not if they're forced to do it. I'm not sure that your vote was the best thing to do then. You also opened the door to unnecessary and bitter comments afterwards.


I think the most likely cause of the conflict is a perceived contradiction between:

  1. justifying the testing process as an act of responsibility and caution; and

  2. choosing to suddenly embark on the process at 4.30pm on a Friday afternoon, which is irresponsible to the extent that people are already tired, haven't had time to plan the work, and will be in a poor position to correct any mistakes they make in the process.

Insofar as the circumstances may have been an emergency, there may also be conflict about whether it is as unusual as a building fire and deserves exceptional attention, or whether it happens frequently enough or is minor enough that it could have been left until Monday morning (or should be covered by a dedicated person who is paid to be happy to routinely work later, or to be on-call).

The test of whether it is a true emergency is whether, had it happened at 1am instead of 4.30pm, would you be ringing telephones and knocking on doors to get all hands in attendance, or would you deal with it later in normal working hours?

So the OP casts the question as whether his choice to test was being too conservative or not. The real issue may be whether he'd already been too cavalier by choosing to do that work at all, rather than schedule it at a more appropriate time.


TBH I think this is more appropriate on say the DBA SE or similar - but here's my thoughts:

If a SQL operation is making a change (Upsert or delete) on multiple records or in Bulk - then it should be tested.

Now, sometimes there's a little lee-way, for example if I was doing a:

update Table

set colA = "Foo"

where colA = "Bar"

Yes, it's an update operation, but since the rollback is as simple as running it in reverse (assuming there are no other values in Col A that equal Bar, then for something like that, I might say 'It's low risk enough that we can forgo testing'.

That said, since it took an Hour of testing, I'm presuming that it wasn't such a simple change as the above.

I often have a similar healthy tension with my Boss, I'm a little more gung-ho, he's a little more cautious and together what we do works.

In terms of your specific situation, given that it took an hour to test, I don't think you were being over cautious, especially on No-Change Friday.

But let me outline some factors and ideas:

1: What was the Dollar Value lost to your client for the extra hour of downtime?
2: What would be the Dollar Value lost to your client if you had to do a full Database restore and then have to fix the problem again, over the weekend, paying OT?
3: How quickly could you roll back the change if something unintended happened? If the roll-back was a 5 minute process, then an hour of testing seems overkill, whereas if roll-back was a 6 hour task then...
4: If the Script fails (either to fix the underlying problem or creates additional issues) - what would be the time required to fix those additional issues?

It's not an easy tightrope to walk, but as a Manager - the above are what should guide your decision. Sometimes the cost to test is greater than the cost to rollback, in which case leniency and expediency is fine. In most cases though, the cost roll back is greater than the cost to test, hence we test.

As the old Adage says - Measure twice, Cut once.

  • Note that even your incredibly trivial example is fraught with danger. What if other systems act on this change, and potentially cache values, etc. Dec 12, 2022 at 7:44

Not the answer you're looking for? Browse other questions tagged .