fetch

A layered web fetcher for agents. One command, escalating through real machinery until it gets a page instead of a wall:

1. yt-dlp, for a video host: the transcript and the metadata, because what is said in a video is not in its HTML.
2. curl_cffi with a browser TLS/HTTP2 fingerprint, replaying the cookies your Waterfox profile actually holds for that host. A connection that fails is retried pinned to the other address family; a 403 is not retried.
3. Zimi at localhost:8899: the ZIM library on this machine answers Wikipedia, the Arch wiki, OSM's tagging reference and devdocs as plain text from a dated snapshot, with no network at all.
4. LibRedirect and IndieWikiBuddy, whose own vendored code picks the alternative front-end they would use and fetches that.
5. Ladder at localhost:8192, which asks as Googlebot.
6. Byparr at localhost:8191, a headless Firefox that solves JS challenges.
7. Camoufox, a headless Firefox whose fingerprint, timezone and locale are generated for each launch from the address the request leaves from. It gets past the gates headless Chromium sits on.
8. Playwright driving system Chromium, for bodies that only exist after JavaScript runs.
9. The page text the browser already extracted.

A path or file:// URL that exists on this disk is not a network target: it skips the chain and goes straight through the text path, which is how a scanned PDF gets OCR'd.

Text out of a binary body is delegated too: pdftotext -layout first, pypdf as a fallback, and pdftoppm plus tesseract when a PDF has no text layer at all. Distillation is the same kind of delegation: readability-lxml for article text, markdownify for Markdown, BeautifulSoup for the pdf.js text layer, NLTK for matching anti-bot wording. Notable images are saved so the model can look at them instead of guessing from alt text, and --shot renders the page in Chromium so you can see the layout too: what is above the fold, how long the page runs.

uv sync
./.venv/bin/python -m camoufox fetch        # the Firefox build Camoufox launches
./.venv/bin/python fetch.py https://example.com/article
./.venv/bin/python fetch.py https://www.reddit.com/r/programming/ --text
./.venv/bin/python fetch.py https://arxiv.org/pdf/1706.03762 --page 1
./.venv/bin/python fetch.py https://en.wikipedia.org/wiki/Chromaticity --shot full
./.venv/bin/python fetch.py https://www.youtube.com/watch?v=jNQXAC9IVRw   # transcript
./.venv/bin/python fetch.py https://wiki.archlinux.org/title/Pacman --tier zim
./.venv/bin/python fetch.py scan.pdf --text                                # OCR a file on disk
./.venv/bin/python fetch.py https://www.reddit.com/ --dns                # what the resolvers say
./.venv/bin/python fetch.py --health

Each run prints one line per tier it attempted on stderr — tier=direct status=403 …, tier=zim … body=mirrored from wikipedia (snapshot 2026-08-25) — so a failure says which tier failed and why.

SKILL.md is the full documentation: what each tier is for, the measurements behind the thresholds and the tier order, and what is known not to work. VENDOR.md records which upstream commits the vendored extension code came from.

Needs

Python	3.13, in a uv venv (uv sync)
Node	with playwright installed globally (NODE_PATH=/usr/lib/node_modules)
Chromium	any binary fetch.py can find, --browser to name one
Camoufox	a pip dependency; downloads its own Firefox build and GeoIP database to ~/.cache/camoufox
Byparr	FlareSolverr-compatible API on localhost:8191 (BYPARR_URL to move it)
Zimi	optional, localhost:8899 (ZIMI_URL), serving a directory of .zim files
Ladder	optional, localhost:8192 (LADDER_URL)
yt-dlp	optional, on PATH, for transcripts
poppler	pdftotext and pdftoppm, for PDF text and for rasterising a scan
tesseract	optional, OCR of a PDF with no text layer, and of an image
doggo	optional, on PATH, for --dns and the resolver checks in --health
Waterfox	optional, profiles under ~/.waterfox (WATERFOX_DIR) for session cookies

Cookies earned by the browser tiers and images saved from pages go under ~/.cache/opencode-fetch/ (OPENCODE_FETCH_CACHE).