Navigating Roadblocks with CI/CD Pipelines and DAST Scanners

Creating and automating a CI/CD pipeline, i.e. improving the process of checking in code, testing it, and deploying it to production, is a goal that more and more organizations have. It is starting to become apparent that security can and should be integrated into such a process, too.

On the surface, integrating your DAST/Active scanner into your pipeline by adding a scan after you've deployed code to a test environment makes a lot of sense. Content providers such as CSO Online have articles explaining why you should get started and vendors such as Probely have published articles talking about how to accomplish it. But there are limitations as to what modern DAST scanners can do that can make useful CI/CD integration difficult. I'll first outline why, then discuss what you can do about it.

Ideal CI/CD Process

Before I start, it might be useful to outline an ideal CI/CD process that utilizes a DAST scanner:

1. Developer checks in code
2. Automated build starts running
3. After build completes, deploy code to test environment
4. Automatically start a DAST scanner
5. If a security vulnerability at or above a certain severity is found:
   1. Stop the process
   2. Automatically create a bug in your bug tracking system
   3. Inform the developer
   4. Continue blocking the build until a fix is in place

After all, most build processes behave in this way. You can fail build processes if unit tests fail, if source code analyzers find a style issue, etc., so why not results from a DAST scanner?

Challenges with DAST Scanners

Good DAST Scans Take Time

If you look at my review of free DAST scanners, you'll notice that aside from ZAP (which I'll get to in a minute), all scans on my relatively small website took at least 58 minutes. Two of the scanners took more than 5 hours to complete their scans. Furthermore, the scans that took longer generally did better at finding results. Arachni, which returned arguably the best results, took over 5 hours. ZAP, which completed in 10 minutes, returned very few results. Some commercial scanners are faster than the free ones, and there are configuration options that can help speed up scans, but any good DAST scan on most non-trivial sites will take at least an hour. This is generally too long for most CI/CD processes.

Not All Results Are Worthy of Your Attention

One of the things I've heard said about DAST scanners is that "because they attack your website, they don't have the problem with false positives that SAST scanners have". This is not my experience. Good DAST scanners will find many security issues but will also churn out a lot of duplicates and false positives. Some scanners will attempt to minimize false positives, but every scanner I've tried of any sort that tried to minimize false positives also had a large number of false negatives (i.e. they whiffed on finding a large number of issues that should have been discoverable). In other words, to give yourself the best chance of finding issues you'll need to deal with a large number of false positives. And you probably don't want a false positive stopping your build.

On top of this, you can expect your DAST scanner to churn out a large number of duplicates, especially header issues that you probably configure once but show up as separate findings for each page of your site. You will likely also want to delay fixing some issues, or even skip fixing some issues entirely. Here again, you probably don't want a build stopped because of an issue you are planning to fix two months from now.

Manual Reviews Are Still Useful

For most scans, there are some findings that are clearly problems that need to be fixed and findings that are clearly false positives that can be ignored. But you will also have a number of findings that need investigation. Sometimes scanners misinterpret what they see and a vulnerability will be miscategorized. Other times a scanner will flag something as "interesting", but you will need to investigate. A very common example of this is what scanners call reflection — which is that a scanner noticed that user input was reflected back on the page, but it couldn't identify a particular vulnerability. Sometimes this finding is a false positive. Sometimes it is a cross-site scripting vulnerability. But someone ought to investigate.

Incidentally, the fact that these issues need to be looked at is a big reason why I like to walk customers through their first few scans. DAST scanners are incredibly useful tools, but are not yet plug-and-play devices for the average development team.

What You Can Do Instead

Just because DAST scanning has drawbacks that make it difficult to use in your CI/CD pipeline does not mean that you should abandon it entirely. DAST scanners are great at finding certain types of issues, and the vast majority of scanners both commercial and free have the ability to be automated. What can you do to incorporate these scanners as a part of your process as smoothly as possible? My recommendation:

• Run the scanner periodically, such as every night or every weekend.
• Make it a part of your process to analyze the results the next day and report findings to the development team as soon as practical.
• Establish SLAs (Service Level Agreements) that the development team will fix all High findings within X days, Medium findings within Y days, etc., so vulnerabilities don't linger forever.

To be most effective, it will be helpful to have a DAST tool that can help you manage duplicates, can highlight new items from the previous scan, etc. Without that ability, managing the list will become too cumbersome and won't get done.

Finally, while DAST scanners aren't good candidates to be run as a part of your build process, some SAST scanners can be. Please contact me to learn more!