19

I've been a developer for over 3 years now and I have built up confidence of my programming abilities within my team. Recently my team has been employed under a company owned by our venture capitalists which we have integrated within and now work on.

I have no experience of Java and while I know how to code in C# etc and I am familiar with APIs, I recently created a huge issue when checking in my code to our source repository.

I was thrown into the deep-end and asked to make changes to an existing API (my team is aware of my exposure and abilities). After understanding the changes and checking the solution environment for references to the API itself, I made the changes, tested it with unit tests and Postman for the response etc.

All seemed fine. However my knowledge of the solution overall and other things is not as good as the other back-end developers on this project; I'm the only full-stack developer, doing both front and back end. The problem was that the API is referenced in a BPMN workflow file which didn't show up when I searched for the references or API name as text in the IDE.

This caused a 4 day problem with between 2-4 back-end developers looking into why the build was failing at any given time during the working day. While I own up to my mistake and fully accept the responsibility for making such a big problem, I now feel very disappointed in myself and more so doubt that I should continue to develop the back-end with such little knowledge of Java and the solution structure. I also can't help but feel guilty for wasting so much time and money on what was a mistake and took a long time find.

While I'd like to be able to spend time and sit down with some back-end developers to go through and understand it fully, they simply don't have the time right now. What else can I do to help ensure I don't make or am far less likely to make mistakes, let alone such costly ones, in the future?

I should add I also had my change code reviewed by one the back-end developers, who I'm sure asked if the API is referenced anywhere else. As my search and check didn't show anything in the IDE as it doesn't check BPMN files, I replied no.

What more and what else should I do to minimize the risk of making similar mistakes in the future?

Additional notes in response to the comments and answers:

Code reviews: I will always sit with the developer and talk through and understand the code with them, explaining as we go.

We do daily scrums. I mentioned this when I was working on it but no concerns were raised, though I'm sure we will discuss it in tomorrows scrum or the next retrospective.

I should clarify that the issue was the build was failing and I understand that I should probably pull all changes into my branch and build and do a final test before committing, this I didn't do.

9
  • 1
    Comments are not for extended discussion; this conversation has been moved to chat.
    – Masked Man
    Jul 20, 2018 at 3:20
  • 1
    Things like this should be documented with the source for exactly this purpose. They aren't always. Did you read the documentation you with reason could be expected to? Jul 20, 2018 at 8:20
  • This kind of sounds like code smell, java like C# is a strongly typed language. If the mechanisms employed for type checking aren't triggered in tests then you may have a deeper problem where interfaces aren't being utilized properly.
    – li x
    Jul 20, 2018 at 8:35
  • This is precisely why code reviews exist. I assume you use source control; why was it so difficult for the devs to review the last dozen or so commits to spot the issue?
    – Alex
    Jul 20, 2018 at 12:00
  • 1
    "a 4 day problem with between 2-4 back-end developers" - So, 8 to 16 back-end developer days were needed to locate the error? Then the issue is not as 'trivial to see and avoid' as some may pretend. If they know 'their' stuff, they can locate the error quickly. If it takes them a couple of days, you're not to blame for not knowing beforehand what was linked with what in the new environment. Unless the issue was clearly documented. But then, why would it take them days to fix it?
    – JimmyB
    Jul 20, 2018 at 13:57

8 Answers 8

35

With multiple developers committing changes to the same code base (and not always being up-to-date with each other), these things will happen. I'm a developer myself; it's nothing to get hung up on, it will happen to all of us at least once. There are a couple of things you can do however to reduce the risk of it happening again.

From the context, it seems as if you tested everything locally and it worked fine. It was only later that something went wrong. This suggests that either your own code base was slightly outdated and/or someone else had outdated code when they committed their own changes. The best thing you can do, just before committing your code, is to pull any recent changes from your repository (or whatever you call it) so you can be confident your changes work with the most up-to-date code. If all your tests pass, then you can commit with a clear conscience.

I would also suggest being a bit more vocal with your colleagues. If you're making changes to a critical system, mention it to them. Even if it's a simple request like "I'm about to change this important bit, do you have time for a code review?". This way, your team won't spend days finding the problem, if there is one, and you can rest easy knowing that you AND your colleagues are confident with the changes. Some development teams do daily stand-ups where they voice what they're working on; it helps the team be more aware of what you're working on and if there's a risk of your changes clashing with anyone else's.

You'll learn by doing. If you are well-versed with languages like C#, you might pick up Java quicker than you think!

3
  • 8
    It was caused by a stringly-typed reference to the API in this "BPMN workflow file", so nothing was outdated and pulling recent changes wouldn't have helped. Jul 20, 2018 at 7:39
  • @DavidConrad you are 100% right there, but this is less about the past and why it happened as im aware and understand how it got to that point. This answer does suggest what i feel has the best advice in this case - using your voice more and getting others involved. Had i been louder and voiced more of what i was doing to other developers it would be more likely someone could have pointed out that it was used in the BPMN workflows. Jul 20, 2018 at 13:37
  • 1
    Ah, the old, "works on my machine." :P
    – Kenneth K.
    Jul 20, 2018 at 13:47
11

This situation happens from time to time - you do something that has unforeseen knock-on effects to something else that you didn't know about before.

The fact that you weren't sacked immediately or put onto a PIP means that the company recognizes that mistakes happen, they get sorted out, and life accordingly moves on.

What can you do about it? Learn about this consequence and see if there's anything else related that you should have taken into account or any other team you should have asked about knock-on effects of you changing an API.

It's all part of building up experience and it shouldn't discourage you going forwards.

6

The way to avoid this problem also provides you with an opportunity to significantly contribute to your new company: the root of this problem is inadequate testing, not you (unless you were negligent in a way that doesn't appear in your post).

In all development environments, it should be possible to figure out if the change you've made is going to cause a major failure before checking in as well as figure out if the code is going to break the build. While it is definitely embarrassing to be the one to find such a gaping whole in the company's process, you now can be a leader in trying to establish such a process at your new company.

You can propose that the team begin writing unit tests and begin setting up a continuous integration environment which would re-build the project and execute the unit tests each time code is checked in, and where the tests can be run individually prior to checking in. The unit tests will take commitment from the team, but a basic CI environment can be set up fairly quickly from open source projects out there as long as you can dedicate a VM or server to it.

Edit: code reviews are another good programming practice that can minimize these kinds of mistakes, and is also fairly easy to get going.

TL;DR use this embarrassing opportunity to modernize your company's development processes. You'll come out smelling like roses.

1
  • 2
    The OP can also use this opportunity to introduce peer review as a SOP, instead of by specifically having to ask for it each time. Peer review can help you learn code, as you read what others do, as well as learn by getting good feedback from other devs on your own code. I'd have to argue that Not all major failures can be caught before checking in. Servers can have some different settings that can break a build that works locally. Simply having NuGet setup differently can cause that kind of problem. Jul 19, 2018 at 19:13
5

The problem arises that the API is referenced in a BPMN workflow file which didnt show when i searched for the references or API name as text in the IDE.

What did the teams retrospective regarding this incident surface? What department procedures could be put into place so there this is not repeated? What does the team believe is necessary when on-boarding new developers? And what new on-boarding procedures are to developed?

P.S. Being proactive in the above is a great way to learn and lead.

7
  • 2
    Can I ask why folks always suggest improvements on the procedural level or management level? It's very doubtful a brand new guy is going to come in and say, "Change all this so my stuff can work!" When everyone/thing has been working "fine" up that point? Breaking something, then not knowing why it broke, take nearly a week of work hours, then at the end say their process stinks and need to be improved is not a great way to start your career there.
    – Dan
    Jul 19, 2018 at 16:38
  • 2
    Because if the question is "What sort of automated testing can we add to the release process that would identify this sort of issue the next time changes are made", you should end up with a better process that doesn't require someone to manually check each change.
    – Gwyn Evans
    Jul 19, 2018 at 20:15
  • Often, procedural and management level problems are ignored by the veterans as they get used to working around them. They forget that things don't have to be that way, so to them things work "fine". It takes a fresh pair of eyes to see the shortcomings. That said, I agree the new guy shouldn't make demands, but their's a big difference between saying "Change all this so my stuff can work!" and "How come we do things this way instead of that way?"
    – Calvin Li
    Jul 19, 2018 at 21:46
  • @Dan Your scenario can be recast: "...then at the end say their process stinks and need to be improved" so as to avoid problems like it in the future. As someone who has done just that, a little dogged determination resulted in process changes which made everyone's lives easier. I swayed my coworkers with around 3 months at the company as a junior developer by doing it and then showing the results. If they aren't doing retrospectives/postmortems that's an easy place to enact positive change that will definitely benefit the team. Jul 19, 2018 at 23:01
  • If a problem occurs, its either through incompetence or process error - it's clear that the OP is not incompetent but made a mistake that is avoidable through some mechanism. The company might not like to hear it from an 'outsider', however this is an issue that proper processes would eliminate. Ideally team management should be able to identify this and apply new processes - if the newbie is initiating this thought process then their are larger concerns. I think the advice of 'proactive'-ness is valid, though I do concede that if the OP is the action-er something larger is wrong. Jul 20, 2018 at 6:21
4

Where were the BPMN files stored that they weren't visible when you were searching for any other API consumers?

The issue sounds like it was that you didn't know about/have access to all of the API consumers.

If they're not part of your standard development working set and for some reason can't be added to it, any APIs that are consumed by then need to be documented as doing so in method call header comments of equivalent. Depending on how formal your review process is explicitly verifying that they're not impacted might be a valid process change.

The issue of your not knowing about everything outside the codebase you're working on that needs to be checked needs brought up at your next standup/retrospective by you if your coworkers don't. Something like that should have been covered as part of your on boarding to the project.

1
  • 2
    I think one of the issues is that IDEs for languages like java (I presume C# is similar but I have no personal experiance) can lull you into a false sense of security. They have built in features for finding the users of classes, methods, fields etc that avoid the false positives found in a simple text search, but those methods are only reliable if all the code is included in the IDE project AND the calls are made directly from java code rather than via reflection. Jul 20, 2018 at 13:28
4

You have to shift your mindset.

You did not make a "critical mistake". You worked to the best of your abilities; you were open to working in an unknown environment; you were transparent about your knowledge and everything else, before and after; you worked together with the other guys to fix the issue, you presumably did not throw a tantrum in the office. After the debacle, you are coming out here and trying to improve even more.

In short: I would lick my fingers to have more employees like you.

To answer your question:

  1. You personally cannot avoid such things, in general. The technical error was completely out of your scope. If you would like to be 100% sure, you would grind to a standstill. You cannot check about parts of a system which you know nothing about.
  2. Systems (including computers and humans) cannot be built to avoid such things, either, as soon as they get complex enough.
  3. One way to approach this is the DevOps/Agile approach of inviting failure, and failing hard and fast. This is what all the CI and expecially CD stuff is about. You deploy so frequently and fast, that you will fail very often; and each time you fix the particular reason with utmost priority (using the usual techniques - automated and manual tests, painless rollback of small commits, blue-green-deploys with snap-of-the-finger rollback-switchover etc.). This logically results in making the whole process more robust very quickly. Ideally you do that from day 0. Easy? No. Can be improved piece-by-piece? Yes.
2

Regardless of what you or anyone does to minimize mistakes, mistakes will happen.

There are, of course, technical "best-practices" that will help and yes, there's all the platitudes about learning from one's mistakes. But you got all that, it seems. Yet you still feel bad enough about the incident that you've taken the time to write it up on workplace.stackexchange.com?

I think there is something tangible that you can do that will help you and the others feel better about this whole thing.

The best thing you can do is to take note of kindness and generosity that was shown to you and express some gratitude for that.

Instead of apologizing even more profusely and making soon-to-be-forgotten "never-again" promises, take the time to personally THANK the folks that helped resolve the problem and those that you've inconvenienced. If it's appropriate bring a box of doughnuts (for example) to the office and complement your coworkers.

And then, the next time something happens and someone else makes a mistake, be nice to them, help them out and return the kindness that was given to you. That's it. You'll be a better team because of things like that.

0

Make your commits as small as possible, and if you can build from the build server after every push. (If it doesn't build often, you may want to build before your push, too!)

This won't help you not make mistakes, but it will make them much less critical. If you know what commit/group of commits caused the build to break it is trivial to revert it while you work on the problem, and if commits are as small as possible then it is easier to locate the source of the problem.

If you are using git, there is even a command to help with this: git-bisect , which will let you do a binary search through a group of commits to find the one causing the problem.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .