0

I am a manager in the cybersecurity division of the company where I work and am responsible for overseeing internal pen testing of APIs and in house built applications. Many applications and APIs are customer facing to be used for customer payments, trade settlement, account management etc. and customers will be entering their personal information into the application (e.g: bank account details).

The problem is that while 2 engineers on the team (out of 10) are good at pen testing tools (e.g: BurpSuite, MetaSploit, Snort, John The Ripper etc.), they are not intuitive or particularly knowledgeable on attack vectors of differing application architectures. One came from an QA background and the other, a development background, but using a different stack. For example, in reviewing their pen test findings report and doing pair pen testing (think pair programming):

  • Common attack vectors that an application using a specific architecture is known to be vulnerable to , are not tested as a test case. E.g: user enumeration attacks in aplications for which credentials have a identifiable pattern.

  • Conversely, attack vectors that are impractical/ infeasible for applications running a selected architecture are tested , wasting time

  • Testing attack vectors on applications that do not meet prerequisites for the attack vectors to be exploitable.

  • Pen test plans that appear to be haphazard with no identifiable plan on which attack vectors to simulate. Plans seems like using all tools and seeing which ones gives results, a shotgun approach far from optimal.

  • Using Kali Linux, I asked my team member why he choose a particular tool, and he answered to see if tool can find a security vulnerability. I was expecting a specific attack vector rather than experimenting with the tool. (E.g: BurpSuite to intercept client / server requests to check user enumeration attacks.). Lack of methodology is concerning.

As a manager, I refuse to sign off on poorly done pen tests that could harm customers. Cybersecurity policy dictates that applications to be used by our customers undergo pen testing prior to PROD release, especially with sensitive data such as bank payment information. He has great knowledge of tools and I have no concerns in any other area, so I do not want to escalate or formally discipline .

How do I coach these 2 team members to improve their "feel" of the attack surface and think like a black hat?

6
  • 4
    If they don't posses the requisite skills, experience, or "intuition" to perform these tests then why were they hired to perform these tests? If they're using the wrong tools or are performing the wrong tests why weren't they trained on using the correct tools and performing the appropriate tests? I'm kind of seeing this as a failure of management, not a failure of these engineers. I wouldn't want to discipline them either for failures that appear to be yours, not theirs.
    – joeqwerty
    Feb 17 at 22:54
  • 1
    @joeqwerty, a bit hard to explain intuition. They know how the tools work , but not when to apply them. They know the syntax of MetaSploit framework , but not when to use this. It's hard to cover the full scope of pen testing when many many use cases are present.
    – Anthony
    Feb 17 at 23:02
  • 5
    a new doctor knows what an ECG does, but not when to apply this or for which patients - Exactly. That knowledge comes from more experienced doctors guiding and mentoring new doctors. If a new doctor fails to develop this "intuition" then that's a problem with their training and guidance.
    – joeqwerty
    Feb 17 at 23:08
  • 3
    ... So pay for expertise, either to do the job or to train these guys or both. Classes exist.
    – keshlam
    Feb 17 at 23:22
  • 8
    What you describe as "intuition" seems to be work experience and know-how with extra steps. Why not call spade a spade?
    – Aida Paul
    Feb 18 at 1:13

2 Answers 2

4

How do I coach these 2 team members to improve their "feel" of the attack surface and think like a black hat?

Simply put: you don't.

If you want your team to deliver high quality, repeatable and consistent penetration tests, it's not good enough to rely on their "feel". What you need is to have a proper structure and methodology that they follow, so that whoever is doing the test will cover all of the key areas, and if they come back and test the same application next year you'll have confidence that they'll still do a good job.

The OWASP Web Security Testing Guide is a great starting point for this. It covers most of the key areas, defines clear objectives for them, and provides some guidance for how to test for that specific issue. It's not perfect, but it should provide a good starting point for you.

Or if you prefer, you can build up your own internal checklists and methodologies, that cover all of the steps you need. And for each area/section, you can require that your tester fills in some comments about what they tested, or why it wasn't relevant - so that you've got a clear record if there's any doubts in future.

This kind of methodology is great for more junior testers, but it's still useful for more experienced ones, because it's easy to forget to check for odd things even if you've done hundreds of tests before. And every time an issue gets missed, you can go back and add it to your checklist so that it won't get missed next time.

And once they've covered off all the key areas in your checklist, then they can start to explore and look at the more unusual or creative areas - but that really just comes down to their experience.

However, it sounds like there's a deeper problem with your team than a lack of methodology.

Plans seems like using all tools and seeing which ones gives results, a shotgun approach far from optimal.
[...]
Using Kali Linux, I asked my team member why he choose a particular tool, and he answered to see if tool can find a security vulnerability.
[...]
They know the syntax of MetaSploit framework , but not when to use this.

These are the most concerning things in what you've posted. What's being demonstrated here isn't a lack of "intuition" or methodology - it's a lack of understanding. You get this kind of approach when a tester doesn't really understand what the issues are, or what they're trying to achieve - they've just learned how to use a load of tools until something works. It's a pretty common thing to see, especially with people who have come into the industry through bootcamps and other short course, and haven't ever learned the fundamentals of how and why stuff actually works.

And ultimately it comes down to a lack of proper in-depth training. You can teach any idiot how to run sqlmap, but they won't have any understanding what it's doing - so they'll miss things and break things. What you need to do is teach them how to do SQL injection, and they're not going to understand that until they understand how SQL works, how applications talk to databases, and what query concatenation is.

So if you want to get your team to the point where they're proper penetration testers, rather than just monkeys throwing all their tools at a problem until something works, you need to go back to basics. And that means taking away the automated tools that do everything for them, and teaching them (or making them learn) how to do things manually.

Whenever they want to use an automated tool for something, they need to answer the questions "What are you trying to achieve?" and "Why do you think that tool is the correct choice?". If they can't give you a decent answer, then they shouldn't be using it.

2
  • 1
    More or less what I needed. Given you mentioned manual techniques, do you recommend I hold off on password attacks where mentoring team mates on automated bruteforcing pen test techniques are common? E.g: hash cracking via Hash Cat or password guessing via JTR / Hydra etc?
    – Anthony
    Feb 19 at 14:26
  • 2
    Offline hash cracking with tools like John or hashcat are pretty safe, and there's not real alternative approach you can use, so I wouldn't be concerned about that. But online attacks with Hydra or Burpsuite Intruder or whatever are much more likely to break things, so there needs to be more thought and understanding before you start using those tools - otherwise it's easy to lock out accounts or cause a DoS.
    – Gh0stFish
    Feb 19 at 14:32
9

You don’t, you provide training and checklists.

“Intuition” the way you are using the word is simply internalized experience. They don’t have this and they don’t need to “use the force,” they need some years on the job using a process that says “hey do user enumeration when there’s auth.”

Or pay a lot more to get people with deep pentest experience already, and they will have the “intuition” you seek.

4
  • 2
    Or pay something between those to send your folks through formal training in the practical application of these tools. Or contract this out. You are complaining about a lack of skill, and you can't mandate skill; you can only develop it or purchase it or hire it.
    – keshlam
    Feb 18 at 5:56
  • 1
    Or, for that matter, set up a series of classes yourself, if you think you know this well enough to teach it clearly rather than just to critique it.
    – keshlam
    Feb 18 at 8:39
  • 1
    As an undergrad I was taught "Good programming comes from experience. Experience comes from bad programming" Same holds true for most fields, including pen testing. Feb 19 at 12:47
  • @LaconicDroid fortunately, no one is a good programmer right away, so experience is basically inevitable. Persistence and a good memory suffice. Feb 20 at 1:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .