Engineering blog
How we build Koji — extraction pipelines, benchmarking methodology, schema design, and lessons from running document AI in production.
-
The Review Queue Is the Product
Most document AI vendors treat human review as the embarrassing fallback for when automation fails. We treat it as the core feature — the part that makes the rest trustworthy.
-
Don't Let the Agent Grade Its Own Homework
An AI agent can improve a document extraction schema on its own — read the failures, edit the config, re-test. The hard part isn't the loop. It's making sure the agent can't lie to itself about whether it worked.
-
HTML Tables Won't Save Your Extraction Accuracy
We tested four table encodings — markdown, HTML, JSON, and CSV — across three models and 232 documents. Re-encoding tables changed accuracy by amounts indistinguishable from noise. The only thing that moved was the token bill.
-
We Don't Trust Our WHERE Clauses
In a multi-tenant system, the worst bug is the query that silently returns another customer's data. Here's how Koji makes that structurally impossible with Postgres row-level security — and the test that proves it holds.
-
Schema TDD: Building Document Extraction Without Opening a Browser
Schema development isn't a configuration task — it's an engineering discipline. Here's how an iterative push-extract-inspect loop gets you to 96% accuracy in hours, not weeks.
-
Bigger Models Don't Extract Better
We tested GPT-4o-mini, GPT-4o, Llama 3 8B, and Llama 3 70B on 165 documents. GPT-4o is worse than GPT-4o-mini at structured extraction — and we found out why.
-
Why Open Source for Document AI
We made Koji open source because the security claims that matter most are the ones you can verify yourself.
-
Where Your Documents Go During Extraction
The first question every security team asks when evaluating document AI: 'If I upload a policy PDF, who sees it?' Here's exactly what happens at every stage.
-
Null Semantics: When "Nothing" Is the Right Answer
Every extraction system can pull values out of documents. The harder problem is knowing when a value isn't there — and handling that correctly.
-
Rate Limits, Retries, and the Hidden Accuracy Killer in LLM Pipelines
We spent weeks investigating a 6% accuracy variance. The root cause wasn't the model or the prompts — it was silent HTTP 429 errors treated as 'field not found.'
-
Why Heuristic Routing Fails on Long Documents
When a 120-page insurance policy goes through extraction, the AI sees fragments. If the router picks the wrong chunks, the AI can't extract what isn't in front of it.
-
Benchmarking Document Extraction: How We Measure Accuracy Across 1,100 Documents
Every document extraction vendor claims 95%+ accuracy. None of them publish how they measure it. We built an open, reproducible benchmark — here's the methodology.
-
Schema-Driven Extraction: Configuration Over Code for Document AI
Most extraction approaches rely on prompt engineering. Schema-driven extraction replaces the hope with a contract — typed fields, validation rules, and routing hints in a YAML file.