> ## Documentation Index > Fetch the complete documentation index at: https://docs.vigolium.com/llms.txt > Use this file to discover all available pages before exploring further. # Spitolas - Browser-Based Web Crawler > State-machine-driven web crawler using real Chromium for discovering application states through user-like interactions. Spitolas is a state-machine-driven web crawler that drives a real Chromium browser to discover application states through user-like interactions (clicking, form filling, iframe traversal). All resulting HTTP traffic is captured at the CDP level and fed into vigolium's scanning pipeline. ## How It Works ``` RunSpider(config, recordSaver) │ ▼ ┌─────────────────────────────────────────────────┐ │ Browser Pool (Chromium via rod) │ │ ├── CDP Network Capture (all tabs/iframes) │ │ └── Auto dialog handler (alert/confirm/prompt) │ └────────────────────┬────────────────────────────┘ │ navigate to target URL ▼ ┌─────────────────────────────────────────────────┐ │ Capture Index State (DOM snapshot + SHA256 ID) │ └────────────────────┬────────────────────────────┘ │ ▼ ┌──────────────────────────────────────────────────┐ │ Crawl Loop │ │ 1. Pick next state (candidates + priority) │ │ 2. Reset browser → navigate to index │ │ 3. Replay shortest path to target state │ │ 4. Fire unfired actions (click/hover/submit) │ │ 5. Capture new DOM → compare → add state/edge │ │ 6. Repeat until termination condition met │ └────────────────────┬─────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────┐ │ Network Writer │ │ CDP events → httpmsg.HttpRequestResponse │ │ → RecordSaver (database) → scanning pipeline │ └─────────────────────────────────────────────────┘ ``` ## Core Concepts ### States and the State Graph A **State** is a DOM snapshot identified by `SHA256(strippedDOM)[:16]`. The **State Graph** is a directed graph where nodes are states and edges are actions that caused transitions. Navigation between states uses Dijkstra's shortest path (with Yen's K-shortest as fallback). **Near-duplicate detection** uses normalized Levenshtein distance (threshold: 10%). For large DOMs (>10K chars), a sampling-based distance is used for performance. ### Actions and Candidate Elements Candidate clickable elements are discovered via CSS selectors (`a`, `button`, `[onclick]`, `[role=button]`, `input[type=submit]`, framework-specific bindings like `[ng-click]`, `[v-on:click]`, etc.). Each candidate becomes an **Eventable** (graph edge) once fired, linking a source state to a target state with an event type (click, hover, enter). ### Fragments (Visual Page Segmentation) Pages are decomposed into **Fragments** - DOM regions identified by XPath, bounding box, subtree size, and content hash. Two modes: * **Landmark** (default): fast DOM-based extraction * **VIPS**: vision-based page segmentation with multi-pass decreasing thresholds Fragment comparison uses the **APTED** tree edit distance algorithm. The Fragment Manager tracks exploration status across duplicate/equivalent fragments and uses **candidate influence** scoring to prioritize which states to explore next. ### Form Handling The Form Handler detects and fills forms with smart value generation: * Field-name-aware values (email, password, phone, URL, etc.) * Constraint-aware generation (respects `pattern`, `min`/`max`, `minlength`/`maxlength`) * Pairwise fallback when filling all inputs at once fails * File upload support with type-aware file selection ### Default-credential login attempts To reach areas hidden behind a login, the crawler can try a short list of common default credentials against a **confirmed local login form** so the browser session stays authenticated and the crawl continues into the now-unlocked area. This pass is **discovery-focused, not a brute-force**, and is heavily constrained: * It runs **only** against a form the crawler has confirmed is a *local* login (a single password field + an identity field + a submit, posting to an **in-scope** host — never an external IdP). * It tries a small, documented default list (`admin:admin`, `admin:123456`, …) plus any identity the crawl registered earlier. The identity field's type is respected (email inputs get email-style defaults). * It is **negative-control gated**: an improbable random pair must be rejected first, or the whole spray is abandoned (so a form that "accepts" anything can't produce a false "logged in"). * It is **single-flighted per host** and capped (≤ 10 attempts, never a wordlist), so it cannot lock accounts. * **No finding is emitted** — the attempts are captured as ordinary traffic and any unlocked pages simply become more crawlable surface. The pass is **off by default** and the [native-scan runner](/native-scan/scanning-modes-overview) auto-enables it by **intensity**: `balanced` uses the minimal list, `deep` uses the full documented list. An ordinary crawl leaves it off, keeping active login attempts explicit. ### Session carry-forward While crawling, the browser often clears an interstitial WAF/bot challenge (a JS or cookie challenge) or picks up session cookies. Those cleared cookies and the browser's User-Agent are **carried forward** into the subsequent discovery and active-scanning phases, so content discovery and module probes reuse the already-cleared session instead of tripping the challenge again. * On by default whenever `--spider` runs; scoped **per host** to the same host the browser visited. Disable with `--no-carry-browser-session`. * Respects your own `-H/--header` overrides — an explicitly supplied header always wins. * The User-Agent is pinned from the browser only when you haven't set a non-default UA yourself. ### Exploration Strategies | Strategy | Description | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Default (BFS/DFS) | Deterministic traversal using fragment-based prioritization | | `adaptive` | **Exp3.1** multi-armed bandit - balances exploitation (known-good actions) with exploration (untried actions) via importance-weighted probability sampling. Rewards based on new state discovery. | ## Browser Management * **Embedded binaries**: ships Chromium (macOS/Windows/Linux) and ungoogled-Chromium (Linux). Extracted on first run, cached by version. * **Headless mode**: uses `headless=new` when extensions are loaded (supports Chrome extensions unlike legacy headless). * **Extensions**: loaded via `--load-extension` (e.g., uBlock Origin Lite for ad blocking during crawl). * **Security flags disabled** for crawling: `--disable-web-security`, `--ignore-certificate-errors`, `--allow-running-insecure-content`. * **Pool**: multiple browser instances with round-robin selection. ## Network Capture Traffic is captured at the **browser level** (not page level) via CDP events, covering all tabs, popups, and iframes: 1. `NetworkRequestWillBeSent` → record request 2. `NetworkResponseReceived` → record response headers 3. `NetworkLoadingFinished` → fetch response body Hash-based deduplication prevents duplicate records. A cleanup loop removes stale pending requests (>15s). Captured traffic is converted to `httpmsg.HttpRequestResponse` and saved via the `RecordSaver` interface with source `"spidering"`. ## Termination Conditions The crawl stops when any of these are met: * Maximum states discovered * Maximum duration elapsed * Maximum crawl depth reached * Maximum consecutive failures * No more candidate actions to explore * Context cancellation ## Entry Point ```go theme={null} result, err := spitolas.RunSpider(ctx, spitolas.SpiderConfig{ TargetURL: "https://example.com", MaxStates: 100, MaxDuration: 10 * time.Minute, MaxDepth: 5, BrowserCount: 1, CrawlStrategy: "adaptive", // or "" for default }, recordSaver) ``` Returns `SpiderResult` with: states discovered, actions executed/failed, forms submitted, duration, and records saved.