Choosing the right Application Security Testing Vendor is not always an easy task. By asking the right questions and knowing what answers to look for, you can conduct the thorough evaluation of the various vendors available in the market and make the most intelligent choice for your business.There are numerous options like buying tools, using cloud based testing providers or the traditional consultants. I have discussed making the right choice in another blog. However, if you decide to choose Application Security Testing consultants, here are the 9 most important questions you should definitely ask based on the top metrics:
1. What’s the background of the individuals who will conduct the test?
The background of the people behind the Application Security Testing is one of the most vital factors. Some companies do have good processes but still the individual plays the most important role. So ask for the background of the people conducting the Application Security Tests.
2. What is the methodology of Application Security Testing?
Though the person is very critical, the methodology of Application Security Testing plays an equally major role. If there is a standard process, it ensures minimal quality irrespective of the state of the mind of the consultant. You don’t want that his breakup with his girlfriend causing a significant reduction in the quality of testing. There should be checks and balances to ensure quality irrespective of the situation.
Different organizations can have different methodology but you need to figure out from methodologies whether key elements like false positives and business logic vulnerabilities are covered.
3. How will he conduct business logic vulnerability testing?
Business Logic Vulnerabilities cannot be detected by scanners. You need very good processes and skills for the Application Security Testing vendor to assess such vulnerabilities. It is important to know how the vendors shall conduct such testing.
4. How will he ensure coverage? Does he have a checklist? Can he share that or show that?
In Application Security Testing, false negatives can create havoc. You also don’t want to miss out a critical business logic vulnerability. So the question is if the consultant has got a standard checklist of tests. Is it possible that you can review the checklist or add certain type of tests to the checklist?
5. What are the contributions of the testers in security research (vulnerability discovery, research papers, tools, conference presentations etc)
Everybody can run a tool. But everybody is not a hacker. You have to fight against the hackers out there on the internet. So it is important that you get a person who matches up to that standard. You should ask him about his background in original security research. Did he do something which is worth being presented in Defcon, Blackhat or other similar conferences?
6. How many and what type of Application Security Tests did he conduct before?
It is important to know the prior experience of the vendor in the field of application security testing. Did he conduct DAST, SAST, Architecture Review, Threat Modeling? You also need to check his experience in discovering Business Logic Vulnerabilities. This is one of the graveyards where many consultants fail unless they have proper experience.
7. Can the vendor test during non-business hours?
Sometimes it might be critical to conduct test during non-business hours (nights/weekends). You need to select a Application Security Testing Vendor who is flexible enough to handle any such requirements that you may have.
8. Can the vendor meet up to your scalability requirements?
The last but not the least; if you have to test all your applications two times as per their respective release cycle or at least on a quarterly basis, will the vendor be able to meet such volume requirements. Do they have the infrastructure and the people to conduct such numbers of application security tests?
Bonus Question: Ask Yourself: Can you conduct adequate number of tests within your current budget using the consultant?
Application security testing is more of a discipline than an event. Doing a great job once in a year will keep you secure only for 1 to 3 months. It is more critical to test during every release. I have written a separate blog on choosing the right frequency of your application security tests. What you need to ask yourself is will the vendor be able to provide you the right quality, frequency and price to meet your needs for the right frequency of application security testing.
Few more suggestions by readers and community members
Credits: Carlos Rodriguez, Milan Danrel
- Customer references, with the ability to interview them. What kinds of problems were found by the vendor, and which ones weren’t?
- Verification of background checks of the individual tester
- Financial statements of the organization.
- Which tools are being used by the tester?
- Integration capabilities to collaborative solutions, GRC solutions, dashboard solutions, QA solutions & ticketing systems.
- Does the vendor meet the compliance specific expectations? (eg. PCI DSS 1.2)?