Hunting The Secret
This started as a sandbox for learning agentic AI. It stopped being a toy the day the decodings began checking out on real ground.
The hunt
In 1982, Byron Preiss buried twelve ceramic casques in parks across North America, then published The Secret: A Treasure Hunt, a book of twelve paintings and twelve verses. Match the right painting to the right verse, decode the pair, and they point to a dig spot; each casque holds a key redeemable for a jewel. Three have been recovered: Grant Park, Chicago (1983); Cleveland's Greek Cultural Garden (2004); and Langone Park, Boston (2019). The other nine have survived four decades of very determined hunters.
The experiment
The honest origin: the treasure was the excuse. I wanted a serious project for learning agentic workflows end to end, covering retrieval pipelines, tool-using agents, provenance handling, multi-model orchestration, and evaluation. Toy datasets teach toy lessons. This problem has real stakes, a fixed ground truth buried in actual dirt, and forty years of messy human data, which makes it a perfect lab: if the system works, there is a hole in a park somewhere that proves it.
And the problem shape is ideal for it. After forty years there is no shortage of clues; they're scattered across the book itself, a couple of expert analyses, community wikis, and thousands of forum posts, with no way to search across all of it at once. So: load everything publicly known into one queryable corpus, and make the matching, decoding, and cross-referencing systematic instead of vibes-based.
The corpus
Everything lands in a local ChromaDB: the full OCR'd book, secondary analysis books, a crawl of the main community site, and forum theory threads. The core of it is three parallel collections of twelve, mirroring how the puzzle actually decomposes: paintings (CLIP image embeddings of each plate, plus reference photos and painting-only notes), poems (verse text with close-reading notes), and puzzles, the assemblies that bind a painting to a poem hypothesis, a city, a park, a dig spot, and a confidence level.
For the nine open casques, the painting↔poem pairing is itself unsolved, so the pairing is treated as a first-class hypothesis rather than an assumption. Alongside the prose notes, a structured clue ledger stores atomic evidence rows (source, observation, interpretation, confidence) so small clues stay retrievable and auditable instead of getting buried in paragraphs.
The agents
A research agent runs a real plan → tool call → observe loop over the corpus: it studies the three solved casques as worked examples, interrogates a painting's notes, tests candidate verses against it, and stages city/park/dig-spot theories with supporting clues. That token-heavy loop runs free on a local Qwen3-32B served OpenAI-compatibly. Full multimodal solve attempts, the ones that need to actually look at the paintings, run on Claude: Opus with extended thinking, and more recently the Fable models. Anything that smells like a breakthrough also gets cross-referenced against OpenAI models before it earns confidence. Agreement across model families counts for more than enthusiasm from one.
The image editor
Each painting gets a workspace in a layered image editor built for this corpus. Plates, reference photos, aerial imagery, and maps stack as overlays with adjustable opacity and pin alignment, so a painting's geometry can be tested directly against a candidate site. Suspected shapes, sightlines, and alignments get traced as vector lines on top of the stack, and the traces stay with the painting's notes. An AI assist panel sends the current view to Claude, an OpenAI model, or both at once, to describe what it sees, compare it against the plate, or argue with whatever the trace is suggesting.
On the ground
Theories eventually have to survive contact with real ground. A phone-friendly field mode rides along on site visits: one-tap buttons write GPS-stamped observations straight into the case file, quick replies log what the site actually shows, and a photo can go to the assistant on the spot for a painting comparison. The confirmation trips so far have all run through it.
Keeping it honest
Systems like this poison themselves the moment a model's guess gets stored next to a fact. Every record here carries a provenance tier (confirmed, generated, or external_unverified), so an agent theory or a forum post can never masquerade as ground truth. Anything interpretive an agent produces is staged into a review queue for human approval before it enters the corpus.
The trust check is a rediscovery eval: run the whole pipeline blind against a casque that's already been found and see whether it re-derives the known solution. If it can't reliably re-find Chicago, its theory about an open casque doesn't deserve much confidence either.
Status
Active, and the most fun I've had with a side project in years. For the first stretch this stayed exactly what it was built to be: a learning exercise. Build the corpus, wire the agents, run the rediscovery evals, and let those evals stay the judge of how much of any "breakthrough" is real signal.
Then the pipeline started producing decodings intriguing enough that leaving them untested felt wrong. So I began making in-person site visits to confirm them, and several checks lined up with features on the actual ground. Results like that are too interesting to ignore. The project has been pushed well past experiment status since: a deeper corpus, sharper agents, and more field trips already on the calendar. Smaller updates land in the log as they happen.