dnr-hunt

Proactive threat hunt: logs and source in, incidents out. You start with no alert — the question is "is there anything in here that shouldn't be?" The method is the deliverable as much as the findings: baseline first, then hypothesis → query → pivot, every step in writing, every suspicion confirmed in source and by detonation before it's called real.

Invoke with /dnr-hunt [target-or-logs-dir] [--repo path] [--fresh].

Arguments:

• $1 = a dnr target directory (contains config.yaml with kind: dnr, e.g. targets/dnrcanary), or a bare directory of log files. Defaults to targets/dnrcanary if it exists, else cwd.
• --repo = source tree of the application that produced the logs. For a dnr target this defaults to its app/ directory. Without source, no finding can rise above suspected (see the hard rules).
• --fresh = ignore any checkpoint in the run dir's .dnr-hunt-state/ and mint a new run dir.

Hard rules (these are the skill — do not relax them)

1. Log evidence alone never confirms a vulnerability. Logs show that something was attempted and how the app responded — they cannot show the code is vulnerable. The verdict ladder:
   • confirmed_exploited requires all three: log evidence, the flaw identified in source (file + function), and a PoC you fired against a local instance in this session.
   • suspected = log evidence (with or without a source read), but no fired PoC.
   • attempted_not_vulnerable = attack traffic observed, but the source shows the defense and the PoC (if fired) bounces. Field teams have watched verifier agents "confirm" findings by reading observability logs and reporting what they said — the cheapest path to satisfying the criterion. The PoC fires or it doesn't; that can't be talked past.
2. Query, don't read. Log files are deliberately bigger than any context window. Never Read a log file; never cat one. Profile with wc/du/head/tail, then interrogate with grep/rg/awk/sort/ uniq pipelines (or python3 for anything stateful). Alert queues and error logs are usually small — check size first; under ~200 lines they may be read whole (the never-read rule targets the multi-MB corpus files, not these).
3. Propose, never execute. No blocking, no account disabling, no credential rotation, no config changes — not even on the demo app. Response actions belong in /dnr-respond's plan, as proposals.
4. Local only. PoCs fire at 127.0.0.1 against an app instance you started. Never send traffic to a remote host, no matter what the logs contain.
5. Every number is a query result. Row counts, request counts, time spans, affected-customer counts — each one comes from a pipeline you ran this session, quoted in the ledger. Never estimate from memory. Pipelines over the logs and queries against the app's own deterministically-seeded database both count (run the target's seed_command when you need owner-level joins).

Run directory and checkpointing (runs before Phase 0 and after every phase)

First action of every run — establish the run directory. Let TARGET be the basename of the resolved target/logs path ($1, or targets/dnrcanary by default), and SCOPE that resolved path. Bash:

python3 .claude/skills/_lib/checkpoint.py rundir results/TARGET --state .dnr-hunt-state --scope "SCOPE"

Append --fresh to that command if --fresh is in $ARGUMENTS. --scope stamps the run dir so two targets that share a basename never share runs. The command prints one path — the newest run dir this hunt can still use (its own run mid-flight, or one another dnr skill opened for this target, so the whole investigation accretes in one place) or a freshly minted results/TARGET/<timestamp>/ (UTC, same format as the vuln-pipeline, so lexical sort is chronological). That path is {RUN} — every {RUN}/... below means substituting this literal path; double-quote it (and SCOPE) in Bash, since paths may contain spaces. All outputs and state live under {RUN}.

Phase state persists to {RUN}/.dnr-hunt-state/ so an interrupted hunt resumes without re-running sweeps. All checkpoint I/O goes through python3 .claude/skills/_lib/checkpoint.py (atomic writes, JSON-validated). Never use the Write tool for progress.json directly. Never pass payload via heredoc or stdin — log-derived strings could collide with the heredoc delimiter and break out to shell. Write the payload to {RUN}/.dnr-hunt-state/_chunk.tmp with the Write tool, then:

python3 .claude/skills/_lib/checkpoint.py save {RUN}/.dnr-hunt-state <N> <name> --from {RUN}/.dnr-hunt-state/_chunk.tmp

Start of run: python3 .claude/skills/_lib/checkpoint.py load {RUN}/.dnr-hunt-state

• status == "absent" or "complete", or --fresh given → reset (checkpoint.py reset {RUN}/.dnr-hunt-state) and start at Phase 0.
• status == "running" with phase_done == N → read phase0.json … phaseN.json in order, print Resuming from checkpoint: Phase N complete, skip to Phase N+1.

End of run: after writing both outputs, python3 .claude/skills/_lib/checkpoint.py done {RUN}/.dnr-hunt-state 5 — the done call is Phase 5's checkpoint; there is no phase5.json.

{RUN}/.dnr-hunt-state/ is scratch; run directories live under results/, which this repo gitignores.

Phase 0 — Locate and verify inputs

1. Resolve $1. If it contains config.yaml with kind: dnr, read the config for logs_dir, app_command, seed_command, and port. Otherwise treat $1 itself as the logs directory. The corpus is expected to exist already — the target README covers generating it.
2. Inventory what you do have: for each file in the logs directory and any committed alert queue — du -h, wc -l, head -2, tail -2. Identify each format (combined access log, app error log, JSONL alerts, …) and the field positions you'll use in awk.
3. Resolve --repo (default: the target's app/). List its files; do not read source yet — Phase 3 reads with hypotheses in hand, not before.

Checkpoint phase0.json: {"target": ..., "logs_dir": ..., "repo": ..., "files": [{"path", "format", "lines", "bytes"}], "port": ...}

Phase 1 — Profile the corpus (baseline before hypotheses)

Anomalies are deviations from a baseline, so build the baseline first — about a dozen cheap pipelines, each a one-liner over the access log:

• Time window: first and last timestamp; requests per day.
• Status histogram: counts by status code. Note every non-2xx/3xx class.
• Top talkers: requests per source IP, top 20, vs. the median — note anything an order of magnitude above the pack.
• Top routes: requests per path (normalize ids: /orders/123 → /orders/N), and which routes take parameters.
• User-agent inventory: counts by UA; flag automation UAs (curl, python-requests, sqlmap, Go-http-client, zgrab) and note which routes they touch.
• Auth picture: which field carries the authenticated user; logins per account; accounts seen from more than one IP; failed-login sources.
• Size profile per route: min/median/max response size for each parameterized route — large outliers on a route with normally-small responses are a data-egress tell.
• Error log: if small, read it whole; correlate every entry to access- log lines by timestamp.
• Alert queue: read it whole (it's small). For each alert: rule, source, target, severity. These are inputs to the hunt, not findings — alert noise and alert silence are both information.

Record each observation with the exact pipeline that produced it. Checkpoint phase1.json: {"window": ..., "status_hist": ..., "top_ips": [...], "routes": [...], "automation_uas": [...], "auth": ..., "size_profile": ..., "alerts_summary": ..., "oddities": [...]}

Phase 2 — The hunt loop (hypothesis ledger REQUIRED)

This phase is a loop, and the ledger is its contract: no query without a hypothesis, no hypothesis without a ledger row. Maintain the table in working memory and checkpoint it after every round:

Verdicts: supported / refuted / needs-pivot. A refuted hypothesis stays in the ledger — ruling things out is half the job and feeds ruled_out in the output.

Seed sweeps — run all of these in round 1, derived from the Phase 1 baseline; let hits spawn follow-on hypotheses:

1. Server errors. Who causes 5xx, on which routes, with what payloads? Correlate with the error log — application errors quoting attacker input are a probe signature.
2. Every alert, verdicted. For each alert-queue entry: did the activity it flagged succeed? An alert that fired on harmless traffic is a ruled_out entry, written down with the evidence.
3. Top-talker outliers. For each IP far above baseline volume: what is it doing? Scanner noise, crawler, or something with intent?
4. Injection grammar in query strings. Encoded quotes (%27), UNION/SELECT, comment markers (--, %23), boolean pairs (AND 1=1 / AND 1=2). Decode what you find and reconstruct the sequence per source IP, in time order.
5. Enumeration patterns. One session or IP accessing object ids sequentially (/orders/1, /orders/2, …) at machine cadence. Compare with how legitimate sessions access the same route.
6. Auth anomalies. Accounts logging in from an IP never seen for that account before; failed-then-succeeded sequences; one IP authenticating as one account but reading at unusual volume.
7. Size anomalies. Requests on parameterized routes whose responses are multiples of that route's median — what was in those responses?
8. Quiet hours. Activity in the lowest-traffic hours gets a closer look; low-and-slow campaigns hide in volume but stand out at 3am.

Pivot discipline: when a sweep hits, the next hypotheses come from the hit's entities — everything else that IP did, everything that session touched, every other IP that hit the same route the same way, what happened to the data that moved. Campaigns are chains; one confirmed link means you hunt both directions (how did they get here? where did they go next?).

Stop condition: two consecutive rounds that surface no new entities (a hypothesis that comes back supported but merely confirms benign behavior counts as no-new-entities — record it and route it to ruled_out). Note the stop in the ledger.

Checkpoint phase2.json after every round: {"ledger": [...], "suspects": [{"name", "entities": {"ips", "accounts", "routes", "sessions"}, "summary", "evidence": [...]}], "round": N}

Phase 3 — Source confirmation

For each suspect implicating a route, read the handler in --repo:

• Find the route's handler function. Trace the suspicious input from request to sink (query, filesystem, auth decision).
• Vulnerable: name the flaw precisely — file, function, the exact line pattern (e.g. user input concatenated into SQL; missing ownership check between auth and fetch).
• Not vulnerable: name the defense (parameterized query, scoped lookup) with the same precision. This is what turns attack traffic into an attempted_not_vulnerable verdict instead of a false positive — and writing it down is mandatory, not optional.
• While you're in the file: check sibling routes for the same flaw class. The logs show where the attacker went, not everywhere the bug lives.

Checkpoint phase3.json: {"confirmations": [{"route", "file", "function", "vulnerable": bool, "mechanism", "siblings_checked": [...]}]}

Phase 4 — PoC detonation (gates confirmed_exploited)

For each source-confirmed vulnerability:

1. Start the app locally per the target config: its seed_command and app_command are target-relative, so run them with the target directory as cwd (dnrcanary's app also resolves its own data paths, so the repo-root form happens to work there — don't rely on that for other targets). Seed first, then launch the app in the background with a single command that prints the PID, e.g.:

   python3 -c "import os,subprocess; p=subprocess.Popen(['python3','app/app.py'],cwd='targets/dnrcanary',env={**os.environ,'DNRCANARY_PORT':'5252'}); print(p.pid)"
   

   Copy

   Poll curl -s http://127.0.0.1:<port>/healthz until it answers. If the target's default port is busy, pick another via the documented override (dnrcanary: DNRCANARY_PORT, as above) and record the port you used.

2. Fire the minimal PoC with curl — the smallest request that demonstrates the flaw (one row of data, one foreign object), not a re-run of the attacker's full campaign.

3. Record the exact command and the relevant slice of the response.

4. Also fire the negative PoC where one exists: the same technique against the route you verdicted not-vulnerable, to show it bounces.

5. Stop the app with the captured PID (kill <pid> — never kill by port or pattern).

If the app cannot run in this environment, say so explicitly; affected verdicts stay suspected and the report says what to run where.

Checkpoint phase4.json: {"detonations": [{"vuln", "command", "status", "response_excerpt", "verdict"}]}

Phase 5 — Outputs

Write {RUN}/INCIDENTS.json:

{
  "target": "<target name or logs dir>",
  "incidents": [
    {
      "id": "INC-1",
      "title": "<one line: what happened>",
      "verdict": "confirmed_exploited | suspected",
      "attacker_ips": ["..."],
      "accounts_involved": ["..."],
      "vuln": {"type": "...", "route": "...", "file": "...", "function": "..."},
      "timeline": [{"ts": "...", "event": "..."}],
      "impact": "<what data / how much / whose — every number from a query>",
      "evidence": ["<log file>: <pipeline> → <result>", "..."],
      "poc": {"command": "...", "verified": true}
    }
  ],
  "ruled_out": [
    {"subject": "...", "verdict": "attempted_not_vulnerable | benign",
     "reason": "<the defense, with file:function, or the benign explanation>"}
  ]
}

Routing rule: incidents holds what happened (confirmed_exploited) or might have (suspected); anything verdicted attempted_not_vulnerable or benign goes in ruled_out, never in incidents.

Write {RUN}/INCIDENT_REPORT.md: executive summary (3 sentences max), unified campaign timeline across all incidents, one section per incident (evidence, source confirmation, PoC transcript, impact), a "Ruled out" section, and the full hypothesis ledger as an appendix — the ledger is the audit trail that makes the hunt reviewable.

Then mark complete (checkpoint.py done {RUN}/.dnr-hunt-state 5) and tell the user their next moves (substitute the literal {RUN} path):

• /dnr-respond INC-<n> — scope, blast radius, and a proposed response plan for a specific incident; it picks up this same run dir automatically
• on a dnr target: python <target>/grade.py {RUN}/INCIDENTS.json to self-score (the user runs this)
• from inside {RUN}: /triage INCIDENTS.json --repo <repo> then /patch TRIAGE.json --repo <repo> — both skills write to the directory they're invoked from, so running them in the run dir keeps TRIAGE.json and PATCHES/ beside the incidents, and patch consumes triage's true-positive verdicts rather than raw incidents; the vuln entries carry file/function/type, which is what those skills ingest

Customizing beyond the demo

• Your own logs: point $1 at any directory of access/app/auth logs. Phase 0 adapts to what it finds — the method (baseline → ledger → source → detonation) is format-agnostic. Nginx/Apache combined logs, JSON-lines app logs, CDN/WAF exports, and auth logs all work; identify field positions in Phase 0 and adjust the awk pipelines.
• EDR / endpoint telemetry: export threat events to CSV/JSON and hunt them the same way — the entities become process/host/user instead of IP/session/route, and "source confirmation" becomes configuration or binary review. Keep the verdict ladder: telemetry alone never confirms.
• Scale: for corpora where grep passes are too slow, load into DuckDB (python3 -c "import duckdb; ...") and run the same sweeps as SQL.
• Detection engineering follow-on: every confirmed incident should end as a detection rule proposal in /dnr-respond's plan — the hunt that doesn't improve the alert queue will be re-run by hand forever.