Ten years of token factories for aviation technical documentation

The part of aviation I worked in was the quiet one. Not the aircraft, not the crews, not the terminals. The documentation — thousands of pages per aircraft family, bound into manuals the maintenance organisation is legally required to keep, update, reference, and prove they have referenced. The sort of corpus that does not appear on the web and cannot be replaced by a vendor's chatbot, because it describes things the vendor has never seen.

I spent about a decade on the technical side of a Nordic aviation company, building and then running the systems that turned that corpus into something an engineer could actually use. The company was later acquired. I can speak on record about the architecture and the outcomes; some vendor and customer specifics stay out of public copy, not for legal reasons, but because they are not the interesting part. The interesting part is what held up across ten years and four generations of language model, and what did not.

This essay is about the delivered work, in first-person, with enough detail that another engineer can recognise the shape. It is the long-form companion to the two delivered use-case pages on this site.

What the work actually looked like

From the outside, aviation technical documentation is a PDF problem. From the inside, it is several different problems braided together.

The primary corpus was the text-heavy side: aircraft maintenance manuals, component manuals, service bulletins, airworthiness directives. The primary output was not a chatbot. It was a retrieval and extraction surface used by maintenance planners and engineers, wired into their tooling. An engineer working on a specific task wanted to know, with citations, which procedure applied, which revision was current, what the required tooling and torque values were, and which related documents they had to cross-check before signing off. "Working on an aircraft," in the office sense, is largely document work.

The secondary corpus was scanned. Decades of back-catalogue pages that had arrived as pixels rather than as a schema: older manuals, manufacturer correspondence, service history. This is what became the scanned-page tagger. The job there was OCR, classification into a taxonomy, and routing into the right digital archive so the retrieval layer could find it. That side ran as a separate pipeline with its own eval harness, and it is the work I would now call "visual parsing with classification stewardship" if I were labelling it today.

Both sides shared the same underlying factory shape. The context engine, prompt compiler, model router, and eval harness were one set of layers, parameterised differently for text and for scans. The fact that they shared a shape is not a retrofit. It was deliberate from the second year onward, once it became clear that maintaining two parallel stacks was a waste, and that the disciplines that made the text pipeline hold up — evals, typed outputs, versioned prompts — were exactly the disciplines the scan pipeline had been missing.

The layers, aviation-specific

The context engine had to handle a peculiarity of aviation manuals: a procedure in one manual routinely references a procedure in another, often by an identifier that is meaningful only within the family. A naive retrieval pipeline returns the cited paragraph; a useful one returns the cited paragraph plus the one that references it, plus the hierarchy above both. Most of the first year of that engagement was spent getting retrieval right. The chunking policy learned to respect section boundaries rather than token counts. The reranker learned that tables are not paragraphs. The eval set grew to several thousand canonical queries drawn from real engineer questions, not synthetic ones.

The prompt compiler started as a set of files and became a small library. Each task type — summarisation, extraction, comparison, procedure lookup — had its own compiled prompt shape, with task-specific exemplars drawn from a reviewed library. The thing that made the compiler worth the investment was the version discipline: a prompt change shipped with a version bump, a rationale, and a pass through the eval harness. When a frontier model deprecation forced a revisit, we were revisiting versions we could reason about, not a single file that had accumulated a year of edits.

The model router was where the swappability paid off. Over the decade we ran, in sequence: a small encoder-decoder for the first extraction tasks, a mid-size open-weights model for the pipeline work, a first-generation frontier API for the hardest reasoning, a switch to a better frontier provider when pricing and quality changed, and eventually a hybrid where self-hosted open-weights models handled routine tasks and a frontier model handled the long tail. The application never knew. The router did. Every time we moved, the migration was a routing change and an eval pass. The migrations were expensive; they would have been much more expensive without the router.

The eval harness is the layer I underestimate every time I describe this work in a single sentence. In an aviation regime, "the system is working" is not an acceptable answer when the system is also making recommendations that feed a process an auditor will eventually inspect. The harness had a canonical eval set, a set of held-out queries drawn from production, a regression gate on every prompt and model change, and — the expensive part — a human review loop for a sampled fraction of outputs. That human review loop was not a nice-to-have. It was the thing that made the rest of the architecture defensible.

What each model generation changed

The most interesting thing about working across four generations of language model is how little the factory had to change, and how specifically it had to change where it did.

The first generation — pre-frontier encoder-decoder models, plus early open-weights LLMs — shaped the decision to invest in retrieval at all. Those models would not read a manual cold. They had short context windows, they hallucinated freely, and they required the system around them to do most of the work. The context engine and the eval harness were born here, because the models could not be trusted to do their own extraction without scaffolding.

The second generation — the first wave of capable frontier APIs — shifted the work inward. Suddenly a large fraction of the extraction tasks could be done by a good prompt, good exemplars, and a capable model. The temptation was to throw away the context engine. We did not. The reason was not conservatism; it was audit. Even when a frontier model could answer a maintenance question from raw text, the system still needed to produce a citation trail, and a citation trail requires a retrieval path. The eval harness also survived, again because of audit: "the model said so" is not an answer to a regulator, and the eval harness was the only way to produce a numeric answer.

The third generation — the long-context frontier models — tested the context engine. For a while it looked plausible that we could stop retrieving and start stuffing whole manuals into the prompt. We ran the experiment. The result was that context-stuffing worked for quality on the easy cases, collapsed on the hard ones, and roughly tripled the cost. More importantly, the citation problem did not go away: an engineer auditing an answer still needed to know which section supported it, and "the model read the whole manual" is not traceable. The context engine stayed. Long context turned into an option the router could reach for on specific task types, not a replacement for retrieval.

The fourth generation — the current frontier, with better reasoning and substantially better tool use — changed what the prompt compiler did more than what the context engine did. More of the logic that used to live in explicit pipeline steps moved into a single prompted reasoning call with tool access. The context engine kept its job of narrowing the material; the compiler learned to produce tool-using prompts; the router learned to budget. The eval harness did the most of the work of confirming that the new shape actually improved outcomes rather than just moving them around.

What did not survive

Several things did not survive a decade of model generations, and they are worth naming.

Task-specific fine-tuning of the language model itself mostly did not survive. There was a period in the middle of the decade when fine-tuning a medium-size model on domain data felt like the right move. It worked; it just aged badly. The frontier moved faster than the fine-tune did, and the fine-tuned model eventually sat in the router as a fallback that was never routed to. The lesson I took from that is narrow: fine-tune the small, cheap model at the end of the router's preference list if it is worth the effort, but do not fine-tune the flagship. The flagship will be deprecated.

Heavily-engineered prompts with long "persona" preambles did not survive. They worked once, in one model, for a specific reason, and they did not transfer. What survived was prompts that were short, specific, declarative, and accompanied by exemplars. The exemplars are the part engineers underweight in their first attempt and cannot live without by their third.

Embedding models that were locked in at provider level did not survive. The embedding space of a production retrieval system becomes a dependency in its own right, and when the provider deprecates the model, the re-embedding cost is real. I now default to an embedding strategy that is easy to re-run from scratch against a new model, and to a scheduled re-embedding against the canonical eval set as a standing operation.

What I would tell a team starting this today

Three things, short.

First: invest in the eval harness before you believe you need it. In the aviation work, the eval harness was the artefact that made every subsequent decision a cheap decision. Without it, each prompt change or model migration becomes an argument; with it, a number. Build the harness on day one, even if the only entries in the eval set are the three questions your first user asks.

Second: treat the context engine as the primary product of the first year. Retrieval quality is the ceiling on everything downstream. Frontier models make retrieval look less important than it is, because they are forgiving; they are not forgiving enough to make up for bad context, and they get stricter every time the auditing conversation tightens.

Third: keep the model behind a router from the first commit. Not because you are going to swap models on day two, but because you will swap them on day three hundred, and by then the codebase has accreted around whatever you started with. The router is the cheapest architectural decision you can make early, and the most expensive one to retrofit.

What a decade produced, in one sentence

A proprietary corpus that had been locked up in bound manuals became a queryable, citation-bearing, auditable piece of operational infrastructure — on the company's own balance sheet, under architecture the company understood, at a cost the company could explain to a regulator and a board. That is the outcome a token factory is for. The aviation engagement is the reason I trust the shape enough to write the rest of the methodology down as a method rather than a hypothesis.