Spitolas - Browser-Based Web Crawler

State-machine-driven web crawler using real Chromium for discovering application states through user-like interactions.

Spitolas is a state-machine-driven web crawler that drives a real Chromium browser to discover application states through user-like interactions (clicking, form filling, iframe traversal). All resulting HTTP traffic is captured at the CDP level and fed into vigolium’s scanning pipeline.

How It Works

RunSpider(config, recordSaver)
  │
  ▼
┌─────────────────────────────────────────────────┐
│  Browser Pool (Chromium via rod)                │
│  ├── CDP Network Capture (all tabs/iframes)     │
│  └── Auto dialog handler (alert/confirm/prompt) │
└────────────────────┬────────────────────────────┘
                     │ navigate to target URL
                     ▼
┌─────────────────────────────────────────────────┐
│  Capture Index State (DOM snapshot + SHA256 ID) │
└────────────────────┬────────────────────────────┘
                     │
                     ▼
┌──────────────────────────────────────────────────┐
│  Crawl Loop                                      │
│  1. Pick next state (candidates + priority)      │
│  2. Reset browser → navigate to index            │
│  3. Replay shortest path to target state         │
│  4. Fire unfired actions (click/hover/submit)    │
│  5. Capture new DOM → compare → add state/edge   │
│  6. Repeat until termination condition met       │
└────────────────────┬─────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────┐
│  Network Writer                                 │
│  CDP events → httpmsg.HttpRequestResponse       │
│  → RecordSaver (database) → scanning pipeline   │
└─────────────────────────────────────────────────┘

Core Concepts

States and the State Graph

A State is a DOM snapshot identified by SHA256(strippedDOM)[:16]. The State Graph is a directed graph where nodes are states and edges are actions that caused transitions. Navigation between states uses Dijkstra’s shortest path (with Yen’s K-shortest as fallback).

Near-duplicate detection uses normalized Levenshtein distance (threshold: 10%). For large DOMs (>10K chars), a sampling-based distance is used for performance.

Actions and Candidate Elements

Candidate clickable elements are discovered via CSS selectors (a, button, [onclick], [role=button], input[type=submit], framework-specific bindings like [ng-click], [v-on:click], etc.). Each candidate becomes an Eventable (graph edge) once fired, linking a source state to a target state with an event type (click, hover, enter).

Fragments (Visual Page Segmentation)

Pages are decomposed into Fragments - DOM regions identified by XPath, bounding box, subtree size, and content hash. Two modes:

• Landmark (default): fast DOM-based extraction
• VIPS: vision-based page segmentation with multi-pass decreasing thresholds

Fragment comparison uses the APTED tree edit distance algorithm. The Fragment Manager tracks exploration status across duplicate/equivalent fragments and uses candidate influence scoring to prioritize which states to explore next.

Form Handling

The Form Handler detects and fills forms with smart value generation:

• Field-name-aware values (email, password, phone, URL, etc.)
• Constraint-aware generation (respects pattern, min/max, minlength/maxlength)
• Pairwise fallback when filling all inputs at once fails
• File upload support with type-aware file selection

Default-credential login attempts

To reach areas hidden behind a login, the crawler can try a short list of common default credentials against a confirmed local login form so the browser session stays authenticated and the crawl continues into the now-unlocked area.

This pass is discovery-focused, not a brute-force, and is heavily constrained:

• It runs only against a form the crawler has confirmed is a local login (a single password field + an identity field + a submit, posting to an in-scope host — never an external IdP).
• It tries a small, documented default list (admin:admin, admin:123456, …) plus any identity the crawl registered earlier. The identity field’s type is respected (email inputs get email-style defaults).
• It is negative-control gated: an improbable random pair must be rejected first, or the whole spray is abandoned (so a form that “accepts” anything can’t produce a false “logged in”).
• It is single-flighted per host and capped (≤ 10 attempts, never a wordlist), so it cannot lock accounts.
• No finding is emitted — the attempts are captured as ordinary traffic and any unlocked pages simply become more crawlable surface.

The pass is off by default and the native-scan runner auto-enables it by intensity: balanced uses the minimal list, deep uses the full documented list. An ordinary crawl leaves it off, keeping active login attempts explicit.

Session carry-forward

While crawling, the browser often clears an interstitial WAF/bot challenge (a JS or cookie challenge) or picks up session cookies. Those cleared cookies and the browser’s User-Agent are carried forward into the subsequent discovery and active-scanning phases, so content discovery and module probes reuse the already-cleared session instead of tripping the challenge again.

• On by default whenever --spider runs; scoped per host to the same host the browser visited. Disable with --no-carry-browser-session.
• Respects your own -H/--header overrides — an explicitly supplied header always wins.
• The User-Agent is pinned from the browser only when you haven’t set a non-default UA yourself.

Exploration Strategies

Strategy	Description
Default (BFS/DFS)	Deterministic traversal using fragment-based prioritization
adaptive	Exp3.1 multi-armed bandit - balances exploitation (known-good actions) with exploration (untried actions) via importance-weighted probability sampling. Rewards based on new state discovery.

Browser Management

• Embedded binaries: ships Chromium (macOS/Windows/Linux) and ungoogled-Chromium (Linux). Extracted on first run, cached by version.
• Headless mode: uses headless=new when extensions are loaded (supports Chrome extensions unlike legacy headless).
• Extensions: loaded via --load-extension (e.g., uBlock Origin Lite for ad blocking during crawl).
• Security flags disabled for crawling: --disable-web-security, --ignore-certificate-errors, --allow-running-insecure-content.
• Pool: multiple browser instances with round-robin selection.

Network Capture

Traffic is captured at the browser level (not page level) via CDP events, covering all tabs, popups, and iframes:

1. NetworkRequestWillBeSent → record request
2. NetworkResponseReceived → record response headers
3. NetworkLoadingFinished → fetch response body

Hash-based deduplication prevents duplicate records. A cleanup loop removes stale pending requests (>15s). Captured traffic is converted to httpmsg.HttpRequestResponse and saved via the RecordSaver interface with source "spidering".

Termination Conditions

The crawl stops when any of these are met:

• Maximum states discovered
• Maximum duration elapsed
• Maximum crawl depth reached
• Maximum consecutive failures
• No more candidate actions to explore
• Context cancellation

Entry Point

result, err := spitolas.RunSpider(ctx, spitolas.SpiderConfig{
    TargetURL:    "https://example.com",
    MaxStates:    100,
    MaxDuration:  10 * time.Minute,
    MaxDepth:     5,
    BrowserCount: 1,
    CrawlStrategy: "adaptive", // or "" for default
}, recordSaver)

Returns SpiderResult with: states discovered, actions executed/failed, forms submitted, duration, and records saved.