Accidentally defaced an internal production website during vulnerability scanning - How do I protect myself

Question

I work in the information security department at my employer. Today while using automated tool of Burp Suite on one of our internal web applications in production to search for vulnerabilities, I accidentally issued a command that defaced the website. If anyone is unaware, certain modules within Burp Suite such as Spider or Intruder, are extremely powerful, allowing fully automated attacks against common vulnerabilities.

Our team is approved by management to conduct security vulnerability scans and approval is documented in a policy document. Proper notifications were sent out before testing began. Another team in company that owns this application questioned me about the unintended changes. I explained the testing was authorized by management of both teams, but during automated scanning, the change they saw was unintended due to a mistake and not attempt to hack / sabotage. They were not happy / convinced and sent email to my manager who is away on vacation.

I am a team lead and have a great reputation with my manager and in the past, successfully delivered several projects in tandem with this other team. I thought about rolling back the changes in PROD , but am afraid this would be an unauthorized change, which it would be strictly speaking.

How do I best explain that my intention was not malicious but a genuine mistake?

**Other than getting change request approved for rollback, how else can I mitigate any fallout?

Given the scale of our applications, manual testing would be inefficient / time consuming. How can I mitigate likelihood of such instance going forward?

Update

I am accepting answer from Machavity as feel both open communication directly with stakeholders and adopting preventative steps modelled from DevOps will most likely prevent future occurrence. The focus on transparency is very helpful!

Answer 1

This sounds like a silo problem

What your question signifies is that your company may be suffering from corporate silos

Organizational silos describe the isolation that occurs when employees or entire departments within an organization do not want to, or do not have the adequate means to share information or knowledge with each other. Siloed teams often end up working in isolation from the rest of the company, leading to a plethora of internal and external problems for employees, executives, partners and customers.

Let's look at your question

Our team is approved by management to conduct security vulnerability scans and approval is documented in a policy document. Proper notifications were sent out before testing began.

Management told your team to test security, so you did. To whom were the notices sent? If your answer is to management, you have a silo problem.

Another team in company that owns this application questioned me about the unintended changes. I explained the testing was authorized by management of both teams, but during automated scanning, the change they saw was unintended due to a mistake and not attempt to hack / sabotage. They were not happy / convinced and sent email to my manager who is away on vacation.

Your silo is security. Theirs is the website. Your silo broke their silo. Heck, they might not have ever heard of you before this incident. It shouldn't surprise you that they're not happy. Someone is going to yell at them for the broken website and they, in turn, will point at you. You will point at management and your edict to test security. At this point management starts to look bad so they'll do some superficial stuff and they'll make it all go away by assuring all involved that they're doing great!

At least until the website breaks again...

Recognize the silos

You need to be proactive here. The first thing is to figure out all the silos. You cannot fix the company infrastructure by yourself. What you need to do is know all the toes you will be stepping on should your silo break another. The last thing you want is another manager yelling at your manager.

In order to do this, start talking to the other teams. Don't do this in an meeting, try to be informal. This is for your benefit only. Chatting other employees up in other teams up can tell you what they're doing. Take note of where your processes intersect theirs.

Defend your job

At this point, frustrated people might go "We don't need that job/", especially if it makes their silo seem more important. Security is one of the first things to go in many corporate environments. Be sure you can explain why your job exists.

Propose to fix your silo

If you've never heard of DevOps before (Chef, Docker, Kubernetes, etc), I suggest you propose it as a solution for next time. DevOps lets you spin up environments on demand, and then you can modify/test/break these spin-up environments without breaking any live ones.

DevOps should be important for mission critical deployment anyways, lest you fail to deploy things correctly and bad things happen.

The events of August 1, 2012 should be a lesson to all development and operations teams. It is not enough to build great software and test it; you also have to ensure it is delivered to market correctly so that your customers get the value you are delivering (and so you don’t bankrupt your company). The engineer(s) who deployed SMARS are not solely to blame here – the process Knight had set up was not appropriate for the risk they were exposed to.

Someone should be able to give you a live environment sandboxed from the real live environment. If not, you're going to break the website again. And again. At this point, short meetings can be beneficial. Make sure you have direct lines of communication to the other silos and do not rely on management to notify them of what your silo is doing.

Answer 2

Rollback your changes.

From the post & the comments it's sounds like you can easily rollback your changes but you do not want to:

thought about rolling back the changes in PROD, but am afraid this would be an unauthorized change

Other than getting change request approved for rollback....

but would this be considered an unauthorized change though? Most likely this will be an emergency change. This rollback was not expected nor explicitly authorized

Note: I'm assuming you took a backup right before starting

Stop playing rules police and rollback your change. If you rolled it back right away there's a good chance no one would have even noticed (but you'd still notify someone so they can make appropriate changes/precautions for next time).

Nobody's going to complain you were authorized to deface the website but unauthorized to revert those changes.

Please get pre-authorized to rollback changes for these kinds of things before they happen. For example, it sounds like making unauthorized changes is some sort of written rule, a clause should be added for rolling back these kinds of mistakes.

Additional note: Security testing sometimes needs to be done on your production environment, not a test environment. Where I am, we always find small edge cases or ways the test environment does not match the production environment exactly, no matter how hard we try.

So although I do hope you have a test environment, I can understand why you'd want to test the production environment. However pen testing should first be done on the test environment, if it fails, you can assume the production environment would too.

Answer 3

How do I best explain that my intention was not malicious but a genuine mistake?

This is your first question.

the change they saw was unintended due to a mistake and not attempt to hack / sabotage. They were not happy / convinced and sent email to my manager who is away on vacation.

They think that you deliberately defaced the website? To what end? Did they not know that the testing was authorized? Are these people the kind to wear tin foil hats?

I accidentally issued a command that defaced the website. If anyone is unaware, certain modules within Burp Suite such as Spider or Intruder, are extremely powerful, allowing fully automated attacks against common vulnerabilities.

From this, I take that the defacement happened because a security vulnerability exists on the production website. How does fixing this not take precedence over the blame game? Why couldn't a random Nigerian Prince deface the website in the same way you did?

Obviously, it would have been ideal to detect the vulnerability without the defacement but this could have happened from a random vandal.

I thought about rolling back the changes in PROD , but am afraid this would be an unauthorized change, which it would be strictly speaking.

What kind of workplace are you in where finding a security vulnerability (even if accidentally) is treated with this level of suspicion? What kind of workplace are you in where you have the capability to do a rollback but don't know whether you are allowed to make that fix? What are the rules if prod goes down? Is nobody on call?

So far I haven't really answered because it seems absurd to need to convince your company that you did not deliberately vandalize the site. If you really need to convince your company of this, the first document you should be updating is your resume.

Beyond that, just gather all the documentation you have about the test being approved and see how much you can shift the conversation to focusing on the security vulnerability. If the culture is the way it seems to be, it may be worth it to talk up the dangers of the vulnerability and why the dev team might need security training.

Given the scale of our applications, manual testing would be inefficient / time consuming. How can I mitigate likelihood of such instance going forward?

Do you not have duplicated test environments? That is where most of this testing should have been done. Clone the existing site and pound away at it with anything knowing that it only exists for security testing. At my job, we have several subdomains which host clones used for various testing.

Answer 4

How can I mitigate likelihood of such instance going forward?

Never test in production unless there is no other viable method. Use a backup system, at least for the initial testing.

If you must test in production, do it during off-hours with a written, approved, and tested roll-back plan in place.

Make sure backups of all data and code are completely up to date and ready to be restored first.

Answer 5

As someone who works in IT Service Management, I'd like to clear up a question you've posed about whether a rollback would be an authorized change -

An issue (defacement of the production website) has occurred that requires an immediate change to be resolved. This is classified as an emergency change request and you should identify and implement the procedure for this.

To be clear, the bug / error / mistake that altered and defaced the website was an unintended / unauthorized change. You now must rollback that change, following whatever your company's emergency change request process is. any larger (250 employee or more) organization will have a clearly defined and established process for this.

Now - Don't go making changes without communicating to the Product Owners - If you've made a change to a website that's "owned" by someone else, you need to keep them in the loop. If they're upset, this is probably the component that is upsetting them... people get stressed out when they're responsible for changes or issues that were created by someone else. The best thing you can do here is to make sure they're aware of what happened, and keep them aware of what you're doing to resolve the issue, expected timelines, etc.

Also, if the change is one that they can make, you should communicate with the product owner of this website, and ask whether they'd like you to resolve the issue (if you have the ability to, through your burp suite app), or if they'd like to work on making the changes required to resolve the issue. Do this very, very politely, without implying that "it's their job" and you're shaking your hands of the situation; Ideally, they'll get the sense that you're keeping them in the loop and appropriately allowing them to make the decisions about what's going to happen with their product.

I wouldn't freak out about it - these things happen. Just communicate honestly and openly with everyone involved, and as long as you weren't seriously doing anything out of line (like trying to do someone else's job to subvert different people's authority or do something you're not supposed to), most of the time you'll be fine.

Now, as far as whether rolling back that defaced website is an approved change or not.... Within Change Management, most organizations have a concept of an Emergency Change Request.

An emergency change is typically a change / issue that requires an IMMEDIATE fix, and in some organizations, may happen with severely reduced, or even no approval beforehand.

The procedure for an emergency change request will depend heavily on your organization - Some may say "Go ahead and make your change to bring systems back online etc and then document with an emergency change request after the fact", and some may say "get your emergency CR in and notify your immediate department's Director [Or person involved in approving emergency changes, usually director level] so they can approve it immediately; If they're not available, notify the director in the next adjacent apartment to get it approved"

This is a good opportunity to become aware of what your departments emergency change management policy is, and how it may differ from the regular change management procedure.

Because the reality is - Things happen. Systems may go down, databases may be corrupt... If you have an issue that's occurred in production unintentionally, and need it resolved immediately, you likely need to implement your organizations emergency change request procedure, as it requires changes that won't fit into the regularly scheduled change request processes and windows.

Ask your manager, or their manager (if they're out of office), what your organizations emergency change management procedure is, and explain what happened. They'll guide you through what to do next.

MOST IMPORTANTLY: Do not hide anything you've done. Do not ever lie about what happened, don't let fear convince you to change the way you would handle this situation. It happens, and 99% of the time, as long as you take reasonable steps to communicate and handle the issue that's occurred, you shouldn't worry about getting fired unless you have really incompetent management or unless the change you've made has had an extremely severe impact on the company.

Be prepared to explain to your manager (which they'll relay to their manager and / or the other department who's application was impacted) why the issue occurred, and what will be done to prevent or reduce the likelyhood of it happening again in the future. There should be a clear action being taken on this; Either a procedure being implemented, knowledge / training being created to prevent people from making this mistake in the future, etc, something to ensure that someone doesn't end up defacing other production websites in the future.

For example - Could this have been done in a test environment instead, without sacrificing the accuracy of the results? Could the button that defaced the website (or the automated script, if it's automated), be disabled? Could an auto-rollback be implemented after it happens and is tested for? These are useful things to identify and to look at to develop a plan for the future to prevent this from occuring again. This is the kind of thing management above you and the other group will be satisfied with hearing after this issue is resolved, to allow them to move back on to other headaches.

Answer 6

As a penetration tester with 5+ years of experience I have a done similar mistakes myself, and seen others do it, so I do believe I have some insights that could be useful.

To address what I take to be the main point from your question is that something bad happened, it had consequences for another team, no one knew quite how to handle it, and both you and the website team ended up a bit confused and flabbergasted by it all. This is actually a "classic" case of mismanaged expectations with regards to what security testing is, the potential negative consequences of it, and poor planning with regards to what to do when something breaks.

Planning better will obviously not undo the damage already done, but this is an excellent opportunity for you and your company to learn how to better prepare for a security test in the future. The website team probably expected your testing to be harmless and without risk, while you expected them to have a resilient website that wasn't easily defaced by automated tools, and to have a way to easily restore to a known good state if something bad did happen.

Having this kind of planning meeting before a security test is usually called a "scoping meeting" in penetration testing lingo, and it is used to prevent these kinds of situations from occurring. I am not sure if you had a meeting with the website team before starting the test, but in the future you should always start a new security test with it so that you can agree on the details of the test and have a common frame of reference. The points below outline what I belive is a must to agree on before starting any security test:

1. Explain what a security test actually is! A lot of confusion can be cleared up by explaining what it is and what you are actually going to do. If you use a word like penetration testing it may not be be in everyone's vocabulary, and it frankly sounds kind of dirty. So explain what it is, why we are doing it, why it is beneficial for them, and perhaps most importantly, what it can not do as it is often seen as a magic ritual that will make sure your application is 100% hacker-proof. Telling them that it is used to discover security vulnerabilities so that the website team can fix them before the evil hackers exploit them usually works good.
2. What is the actual scope of the test? Is there anything that is strictly off-limits and should not be touched? Note it down, and make sure all parties have it in writing and agrees to it.
3. What kind of environment should the test happen in? You did testing in production, and sometimes there are no other options. But if you are testing in a high-stakes environment make sure that all parties are aware of it and agrees that it shall happen there, and that they are aware that the worst-case scenarios could include loss of data and/or its integrity. This is often the place where you need to ask the "obvious" questions such as: "Do you have a backup?" and "What will you do if I break something on your website?". If they can't answer these questions in a reassuring way it is perhaps time to think about whether or not they truly are ready for security test. If your planned and controlled security test can bring them into a non-recoverable state your company most likely have bigger issues, and you should look at resolving this before proceeding with such a high-risk test. If they have them, make sure its documented and agreed on that they have a responsibility to bring things back if something breaks in the test.
4. If you are an american and/or external party, make sure that all the necessary legal paperwork is created and signed. Have your legal department help you as there may be interesting legal pitfalls, such as parent company vs. sub-company obligations, parts of the servers in other countries with different laws, personal liability questions, etc. This is to protect both you personally from any claims, and to protect the company from itself.
5. Agree on how, who, when and what for giving information about the starting, stopping, and progress of the test. Some people like to be informed when you start testing and against what so that they can keep an extra eye on what's going on. Giving information is always crucial so you don't give the impression of having "gone rogue".

The above is not an exhaustive list, but I believe it can help you going forward to avoid getting into similar situations again. Security testing is unfortunately inherently dangerous as it is at its core about proving that a computer system can do something it was not intended for, while the rest of computing is mostly interested in demonstrating that computers can do what they were intended to. And don't let this bad experience discourage you or your company from doing future security testing. Better planning will minimize the risk, and as with anything new there is always growing pains, but over time you will demonstrate value and you will be seen as a benefit and not just a new potential security incident.