From claim packet to auditable action

Context and stakes

Claim packets arrive as collections, not clean records. Page identity, duplicated documents, missing context, and conflicting values make a model response only the beginning of the engineering problem.

The public goal is not to reproduce a private implementation. It is to show the control structure required when an AI-derived field can affect real operational work.

My role

I worked across orchestration, document understanding, structured extraction, validation, reviewer routing, evaluation, and telemetry. The central design decision was to represent every step as an explicit state transition rather than a chain of opaque model calls.

Constraints

Regulated data could not appear in public artifacts. Model confidence was not accepted as a business rule. Latency and cost mattered, but correctness, traceability, and recoverability came first.

• Mixed multi-document inputs
• Policy and procedure context
• Partial or contradictory evidence
• Human accountability for unresolved risk

Architecture

A packet manifest establishes page identity and provenance. Classification selects a typed extraction contract. Deterministic checks, cross-document consistency rules, and confidence gates decide whether the result can proceed or must enter review.

The audit trace records the contract version, tool events, validation results, reviewer route, latency, and final disposition without storing sensitive source content in telemetry.

Key decisions and rejected alternatives

I rejected free-form JSON that downstream code would have to interpret. Versioned schemas made missing fields, invalid enums, and incompatible changes visible before handoff.

I also rejected a single pass/fail confidence threshold. Field-level uncertainty, business-rule failures, and missing evidence require different reviewer queues and different remediation.

What failed or was unreliable

OCR was least reliable around page identity, section boundaries, low-quality scans, and repeated headers. Model-generated confidence was not calibrated enough to stand alone. Those failures drove the manifest, deterministic validators, and reviewer fallback.

Validation, review, and operations

Evaluation separates extraction accuracy, validation coverage, routing accuracy, and workflow outcome. Reviewers see the source reference, normalized value, reason for the route, and the failed rule—not just a red badge.

Operational monitoring tracks stage latency, retry patterns, tool failures, validation rates, review volume, and drift in document mix.

Outcome

In a measured claims workstream, the approach supported an approximate 90% reduction in document-handling effort and an approximate 50% reduction in time-to-claim-payable. These are scoped, résumé-supported outcomes, not enterprise-wide promises.

What I would improve next

I would expand slice-based evaluation by document family, add reviewer-agreement analysis, and formalize a shadow-mode release gate before any new extraction contract receives downstream authority.