Burp Scanner can now crawl static sites between 6x and 9x faster | Blog


Burp Suite Professional version 2022.2.3 made the Burp Scanner crawler between 6x and 9x faster when used against static or stateless sites. This helps you perform automated recognition much faster than before. Be sure to use the latest version of Burp Suite Professional – and select the “Fastest” scan strategy when creating a new Burp Scanner task, to see it in action.

Burp Scanner fastest crawling strategy - illustrated in Burp Suite Professional

To access Burp Scanner crawl strategies in Burp Suite Professional, go to Dashboardclick it New scan button, then click Scan configuration > New … > Crawling > Crawl optimization > Crawl strategy.

Crawl? What is that?

When we released Burp Suite 2.0 in 2018, it brought many changes. Not the least of these was the replacement of the venerable Spider Tool with an all-new crawler. This meant that for the first time, the recognition phase of tests could be fully integrated into Burp Scanner.

The crawler is much better suited to the demands of the modern web than the Spider ever was. It’s smart enough to navigate and perform reconnaissance even on today’s dynamic, stateful, and/or JavaScript-heavy applications – and we’re continually developing those capabilities.

We would like to introduce you to the new and improved exploration strategy “The Fastest”. You could call it an “incy wincy” version of the crawler…

The problem of overthinking

The problem is that not all of the web applications you might use Burp Suite to test are built on cutting-edge technology. There are plenty of sites out there that don’t have any of the complex features discussed above, and instead settle for good old-fashioned static HTML.

Static HTML sites are not particularly complicated – but performing manual recognition on these basic applications is still time consuming. Although Burp’s crawler can help you at this point by automatically navigating to your target site, in the past it wasn’t well suited for simplistic static applications of this type. In fact, the problem was that the crawler was too smart for its own good – it was overthinking.

In testing, we saw crawl speeds improve from 6x to 9x

Because of this, some Burp users have noted that the crawler isn’t always faster than the original Spider when crawling static sites. While Burp Scanner’s “Fastest” crawl strategy was optimized for applications lacking stateful functionality, we knew there was a lot more performance to be had in this context.

As part of the Burp Suite 2022 roadmap, we looked at different ways to improve crawl speeds – and as such, we worked on improving the above issue. So we would like to introduce you to the new improved and lean exploration strategy “The Fastest”. You could call it an “incy wincy” version of the crawler…

If you’ve read our in-depth analysis, “Web Application Mapping: Mapping the Burp Suite Crawler” by PortSwigger Scanner Engineer Tom Shelton-Lefley, you’ll know that automatically navigating through a modern, complex web application is not no small feat. For this reason, the crawler has to do quite heavy work.

By analyzing the actions of the crawler on basic static sites, we could see one process in particular opposing faster crawling. It was the process of allowing Burp’s crawler to navigate stateful applications – where being aware of the path it took to get to a page is important – but not required when state is n is not a problem.

The stateful application architecture means that a change in the state of one page can easily change what you find when you open another page. Adding items to a shopping cart is a good example of this status change. This means that for a stateful application, Burp’s crawler must be aware of the actions it took to arrive at a page in a certain state.

But this problem becomes irrelevant when dealing with a non-stateful application. This meant that we could remove this process from Burp Scanner’s “Fastest” crawl strategy. And because Burp Scanner no longer cares about an application’s state, it’s able to crawl and scan it for vulnerabilities much faster – making fewer requests in the process.

Benchmarking the new strategy – how much faster is it?

The Burp Suite documentation is a great example of completely static content. Although there is a wealth of knowledge contained within its hallowed pages, there is certainly nothing groundbreaking in terms of functionality. For this reason, it is an excellent benchmark against which to measure the new “Fastest” crawl strategy.

When you do this, the results are simply stunning. In testing, we saw crawl speeds improve by 6x to 9x, depending on whether Burp’s browser is enabled or not. (Burp’s browser enables application crawling where pages are built on the client-side using JavaScript – see our in-depth analysis of browser-based crawling for more details.)

In short: using the improved “Fastest” strategy

Burp Scanner offers five exploration strategies for you to choose from – ranging from “fastest” to “most comprehensive”. Broadly speaking, these range from being good for completely static sites without any stateful functionality (“Fastest”), to being better suited for complex apps that are heavily stateful, including modern d only one page (“The most complete”).

If you were to use the “fastest” crawl strategy on a single-page application written using a library like React, you’d probably be disappointed – because it would find little attack surface to test. Similarly, if you were to use the “Most Complete” strategy on a completely static site with no stateful functionality, you would find yourself waiting much longer than necessary for the crawl to complete.

The key is the context. By selecting the right scanning strategy for your target application, you can maximize the additional value that Burp Suite Professional enables you to deliver – finding more vulnerabilities, faster. For more information, see our Burp Scanner exploration strategies documentation – or see our guide to running your first Burp Suite Pro scan if you’re new to Burp Scanner.


Comments are closed.