How We Build Regulated AI
Eight engineering principles — distilled from production AI systems built under FDA contracts, EU MDR regulatory constraints, and 38 CFR compliance requirements.
Every Layer Is a Deliberate Choice
Each stage of the pipeline answers a specific question: What is the evidence? What does the rule require? What is genuinely uncertain? How confident are we? Who reviewed it? How do we know it is still working?
Evidence Sources
FDA · PubMed · ClinicalTrials.gov · EUDAMED · Medical Records
Regulatory Logic
38 CFR · EU MDR Classes · FDA 510(k)/PMA · VASRD
Evidence Retrieval
Provenance tracked · Authority-weighted · Pre-filtered for relevanceSource-authority-weighted retrieval from FDA, PubMed, ClinicalTrials.gov, regulatory registries, and domain-specific corpora
Deterministic Guardrails
Zero-LLM for rule-bound determinations · Confidence 1.0 where mandatedRegulatory rules applied as algorithmic logic — when rules are unambiguous, no LLM is involved
Bounded LLM Reasoning
Output schema enforced · Confidence bounded · SME overrides blocked from rewritingLLMs handle genuinely ambiguous tasks only — output-constrained, evidence-anchored, with confidence caps and schema validation
Structured Output + Provenance
Per-output attribution · Confidence decomposed · Audit trail storedEvery output carries source links, per-component confidence scores, and the evidence chain that supports the claim
Human Review Integration
Overrides stored with rationale · Analyst decisions improve coverage · Expert judgment preservedAnalyst workflows, SME override paths, and accept/reject feedback that routes back into the pipeline and improves future runs
Production Monitoring
Golden cohort gates deployments · Cost per run tracked · Drift detected before it becomes an incidentGolden cohort regression gating, statistical drift detection, and cost instrumentation — deployed and measured from day one
Deterministic-First
“Where rules are unambiguous, use no LLM.”
Regulatory rules — FDA risk classes, VASRD rating schedules, EU MDR classifications, coverage determinations — are not ambiguous. When statutory or regulatory criteria are met, the outcome is mandated. Using probabilistic inference where deterministic logic is required introduces failure modes that accuracy benchmarks cannot detect. We implement unambiguous regulatory logic as executable algorithmic guardrails, not model outputs. LLMs handle the genuinely ambiguous parts: interpreting narrative medical records, reasoning across biomedical knowledge, synthesizing multi-source evidence.
Evidence from production
Applied: Presumptive Nexus Engine — 20 classification rules across 7 legislative categories, zero false positives, confidence 1.0 by regulatory design. Applied: VASRD combined rating formula (38 CFR §4.25) — LLM extracts severity descriptors; deterministic lookup produces the rating percentage.
Evidence-Grounded Retrieval
“Every LLM call is anchored to specific evidence, not general knowledge.”
In regulated environments, 'the model's training data suggested' is not an acceptable evidentiary basis. Before any LLM reasoning step, we retrieve the specific documents, filings, registry records, or clinical studies relevant to that determination. LLM calls are made against that evidence — not against the model's general knowledge. We apply source authority weighting to distinguish regulatory sources (FDA, EU MDR) from academic publications from manufacturer claims, and we track provenance so every output can be traced to its exact evidence chain.
Evidence from production
Applied: ECRI Time-to-Market — 6-tier authority weighting: REGULATORY=1.0, ACADEMIC/CLINICAL=0.8, MANUFACTURER=0.5, NEWS=0.2, UNKNOWN=0.05. Applied: Trajector Direct Nexus — 99.4% of nexus opinions link back to specific medical records in the veteran's own file.
Bounded LLM Inference
“Output constraints are a design requirement, not an afterthought.”
We do not ask LLMs to produce final regulated outputs directly. We define bounded output schemas, cap confidence where the evidence doesn't support certainty, block LLMs from overwriting human-set decisions, and require explicit evidence gates before inference fires. In high-stakes pipelines, the failure mode of an unconstrained LLM is not a wrong answer — it is a confident wrong answer that looks indistinguishable from a correct one. Structured output schemas, schema validation, and retry-with-context patterns are standard in every pipeline we build.
Evidence from production
Applied: ECRI TTM adjudicator — Claude proposes intermediate phase labels only, confidence hard-capped at 0.5, blocked from overwriting SME-set class decisions. Applied: VA Rating — LLM extracts severity descriptors from narrative; VASRD table produces the rating (LLM never outputs a disability percentage).
Human Review by Design
“AI is a force multiplier for expert judgment — not a replacement for it.”
We design the human review workflow before we design the pipeline. In regulated environments, expert analysts, clinicians, and regulators will review AI outputs — and they will override them when they're wrong. We build systems where those overrides are tracked, stored with rationale, and routed back into the pipeline. Analyst decisions improve the system; they don't just patch individual outputs. The goal is to maximize analyst throughput by reducing what requires their attention — not to eliminate their review.
Evidence from production
Applied: ECRI Horizon Scanning — 5,000 weekly device signals reduced to ~100 for analyst review, with zero false negatives on known novel approvals. SME overrides stored in PostgreSQL with timestamps and decision reasoning. Applied: ECRI TTM — 122 SME hard overrides; SME override API; system confidence thresholds only raised after SME accuracy validation.
Golden Cohort Evaluation
“If you can't measure it, you can't deploy it.”
Every production AI system we build ships with a structured evaluation harness: a golden cohort of known-outcome cases that gates every deployment, automated quality checks per production run, and LLM-as-judge evaluation for outputs where ground truth is expert judgment. We instrument cost, latency, coverage, and quality as first-class metrics — not afterthoughts. We define what 'good' looks like before writing the first pipeline stage, and we track whether the system is getting better or worse with every change.
Evidence from production
Applied: Trajector VAS — 9-veteran golden cohort with known ground-truth outcomes gates every production deployment; 11 automated QA checks per run; 253+ tests across the platform. Applied: ECRI EMBASE — model selection (Opus over Sonnet) made on calibration quality rather than raw accuracy — appropriate uncertainty expression over overcalling.
Lab → Production Discipline
“Prove it in the lab. Then port it exactly.”
We build and iterate AI pipelines in a dedicated R&D environment before shipping to production. Production code mirrors the lab one-to-one; automated sync-checks police drift. This discipline matters in regulated environments because changes that look like improvements in isolation can introduce unexpected behavior at the system level. The rule: iterate freely in the lab; port only proven behavior to production; never skip the validation step because you're in a hurry.
Evidence from production
Applied: ECRI — all five pipelines developed through dedicated R&D phases with staged promotion. The TTM system went through 77 iterative improvement phases before reaching current production accuracy.
Provenance as Architecture
“Tracing outputs back to their exact origin — not just inputs forward.”
Evidence-grounded retrieval governs what goes into an LLM call. Provenance governs what comes out. For every consequential output, we design the full backward chain: final attribute → specific evidence text → source URL/document → retrieval timestamp → model + version + prompt → specific pipeline run. This is the difference between 'the model said so' and 'here is the exact evidence and exact process that produced this conclusion.' Provenance and traceability are related but distinct: provenance answers where a conclusion came from; traceability reconstructs what happened to the evidence throughout the system. Both are required in regulated environments.
Evidence from production
Applied: ECRI Attribute Validation — 45 attributes per device, each with URL audit trail, retrieval timestamp, and authority score. Applied: Trajector Direct Nexus — 99.4% of nexus opinions (171/172) link to specific medical records in the veteran's own file, not to model training data. A reviewer can identify the exact document sentence that supports each service-connection conclusion.
Production Observability
“Every inference leaves a traceable record. Not just logs — structured audit trails.”
Every LLM call in production generates a structured trace: model, temperature, prompt version, output, token counts, latency, cost, and provenance metadata. This is not aggregate logging — it is per-inference record-keeping that lets you reconstruct what happened for any specific output, detect model drift before it becomes an incident, measure cost-per-run as a first-class metric, and prove to a regulator or auditor that a specific conclusion was produced by a specific model at a specific time under a specific prompt. We instrument observability from the first pipeline stage, not as an afterthought. Privacy-preserving tracing design is an explicit architectural requirement: content tracing is off by default for sensitive document pipelines; metadata-only traces are sufficient for most observability purposes.
Evidence from production
Applied: ECRI — MLflow tracks every pipeline run; cost, token counts, coverage, and block-reason breakdowns are first-class metrics per run. Applied: Trajector OL Use Cases — 11 automated QA checks per production run: evidence quality, grounding completeness, nexus opinion traceability, rating calculation accuracy, cost baselines, and latency thresholds — every deployment validated before reaching advocates.
See this in practice
The ECRI Medical Device Intelligence Platform is the reference case — five production pipelines, active FDA contract, and every principle above applied at scale.