# Changelog

All notable changes to **robotstxt-tester** are documented here.
Format: [Keep a Changelog](https://keepachangelog.com/).

## [Unreleased]

## [0.1.0] — 2026-11-15

### Added

- **Initial scaffold** of `robotstxt.btoven.net` — the robots.txt tester of the
  BTOVEN fleet (website docs/service-transparency.md §11, decision O2:
  *consultar y reportar*, never block).
- **`robots` crate** — the consumable module other checkers import instead of
  each fetching robots.txt on their own: `fetch` (origin-based, UA
  `BTOVEN-Robots/1.0 (+https://btoven.net/transparency)`, 500 KiB / UTF-8 /
  bounded stream read) + `parse` (RFC 9309 groups, longest-match-wins with
  Allow-on-tie, `*` wildcard, `$` anchor, global `Sitemap`, unknown directives
  recorded) + `AgentDecision` (what the file says about *our* agent) with
  `allows(path)` for programmatic use.
- **Per-origin cache (§11.3)** — key = origin (`scheme://host[:port]`), TTL
  policy with anti-recrawle floor: 6h floor / 2 days default / 30 days ceiling,
  honoring the robots.txt `Cache-Control` (`cache-control`), stable absence
  (`absent`), errors in the exclusion window (`floor-error`). Configurable via
  `BT_ROBOTS_TTL_FLOOR_SECS` / `BT_ROBOTS_TTL_SECS` / `BT_ROBOTS_TTL_MAX_SECS`.
- **L3 shared cache** (`shared_cache.rs`, CACHE-001 pattern) — Redis
  (`BT_CACHE_URL`) with **per-entry TTLs** and distributed single-flight: one
  fetch per origin across processes/servers; degrades to L1 + direct fetch when
  Redis is down. Namespaces `btoven:robots:lookup:*` / `btoven:robots:lock:*`.
- **Transparency surface** — every JSON response carries the public cache state
  (`cache.key` origin, `ttl_secs`, `expires_at`, `policy`) for the fichas at
  btoven.net/transparency; UA + registry signature
  `BTOVEN-Robots/1.0 (+https://btoven.net/transparency)`.
- **MCP consumer contract** (B1–B4, B7): `/health`, `engine=robotstxt`,
  `schema_version=1`, error envelope, polarity over `fetch_errors` /
  `validation_errors`; `llms.txt` with contract + privacy + cache policy.
- **SSRF guard before fetch** (SSRF-001): `contract::guard_target` → 400 +
  `{error:'private_target', retry:'never'}` for RFC1918/loopback/link-local/
  cloud-metadata targets on both `/api/lookup` and the web route.
- **UI** on the shared theme base (TEMPLATE-001): RFC 9309 intro with the
  *signal, not control* stance, agent-decision panel (match badge + sample-path
  verdicts), rule-groups table, sitemaps, raw content, and the *robots cache*
  panel. NO-JS friendly (server-rendered; the only script is the theme toggle).