Positive Technologies engineers have shared how they built a dynamic crawler for the PT BlackBox security scanner to handle modern single-page applications. The scanner receives an application URL and returns a list of vulnerabilities after black-box analysis. The crawler component collects the attack surface by discovering endpoints, methods, and parameters before vulnerability checks begin.
Static crawlers that only render pages and recursively extract links from the initial DOM are insufficient for SPA interfaces. In a typical React or Vue personal cabinet, clicking a profile picture reveals a submenu, settings open new tabs, and profile photo selection triggers background requests that never alter the URL. None of these elements exist in the original HTML document; they appear only after prior interactions.
The team defines a static application as one where all endpoints can be collected without interface interaction. A dynamic application requires clicks, scrolls, or text input to reveal additional endpoints and parameters. The dynamic crawler must therefore operate directly on browser state to answer two questions: which elements on the page can be interacted with, and what changes after each interaction.
Interactive elements are hard to identify from HTML alone. Handlers may be attached via addEventListener, delegated through framework root containers, or hidden inside closed shadow DOM. Conversely, visually prominent buttons may be inactive or blocked. After an interaction, the crawler must decide whether a new state has been reached by comparing observations, because URL and network activity often remain unchanged.
States are represented as sets of 64-bit fingerprints of currently enabled interactive elements. Two states are equivalent exactly when their fingerprint sets match. The URL is stored only as a possible navigation hint and does not participate in equivalence. This definition yields an equivalence relation that is reflexive, symmetric, transitive, deterministic from external page observations, and cheap to compare via hash lookup.
The crawler therefore explores the factor set of states rather than every reachable state. This keeps the number of visited classes finite and growing slower than the number of performed actions, satisfying the requirements of reproducibility, autonomy, termination, and speed needed for vulnerability scanning.
