Multi-Format Health Data Intake, Validation and Conversion Platform

The Business and Technical Challenge

A state public-sector health and social-care initiative required dozens of independent senders to submit social-needs screening and consent data in a shape its downstream data lake would accept, but nobody could observe data quality per sender until the destination lake rejected a bundle.

• Senders differed widely in tooling, transmission schedules, and data formats (native FHIR JSON, flat CSV archives, clinical C-CDA XML, and MLLP HL7 v2 feeds).
• The platform had to provide each sender fast, readable validation feedback before data reached the state data lake.
• The platform had to accept non-FHIR input (CSV, C-CDA, HL7 v2) and convert it into conforming FHIR R4 bundles.
• Each sender's data had to remain strictly separate and attributable to a specific tenant.
• Every incoming request and outbound response had to be recorded so missing or rejected bundles could be traced, investigated, and replayed.
• Implementation Guide releases and code-set packages were controlled by an external state entity, requiring frequent schema updates without disrupting senders already live in production.
• The platform had to extend to managed-care organizations (MCOs) needing outbound FHIR deliveries and monitored file drops.
• Different FHIR validators flagged different items, requiring all testing to execute against the production validator.
• Sensitive data boundaries required explicit security labels on substance-use (Part 2), mental health, and developmental disability markers, while extraneous clinical sections (allergies, medications, labs, vitals) required data minimization prior to validation.
• Lower testing environments strictly prohibited raw identifiable patient data, which required senders to supply de-identified test samples and slowed onboarding verification cycles.
• Senders could not always modify their source systems, requiring the ingestion gateway to compensate for missing fields via interface contracts.
• Other external parties tested in the shared QA environment, meaning unreleased Implementation Guide work could not be promoted there prematurely.
• Cloud service quotas forced architectural adjustments: load-balancer quotas on per-port routing, client-certificate chain length limits (where three intermediate certificates passed but four failed), and default payload size caps (1 to 2 MB limits against 4 MB incoming archives).
• Production promotion, cloud accounts, DNS, and identity administration remained with the program operator; engineering access was granted case by case and revoked after testing.

Ingestion Categories and Heterogeneous Sender Handling

Seven distinct categories of input arrived across the platform, each presenting specific technical failure modes that required dedicated handling:

• Native FHIR: Received as FHIR R4 JSON bundles over HTTPS from regional data-sharing organizations. Bundles were required to declare an official Implementation Guide profile URL, and some senders omitted bundle identifiers. Bundles were validated and never modified, with missing profile URLs rejected on arrival.
• Flat Files: Single CSV files and compressed zip archives transmitted over HTTPS and SFTP by regional organizations and community providers. Senders used differing specification versions, large zips caused synchronous 5-minute HTTP timeouts, and files arrived with invalid suffixes, missing patient IDs, or illegal null characters. The platform applied Frictionless schema validation, asynchronous queue processing, row-to-FHIR conversion, and readable error reporting.
• Clinical Documents: C-CDA and CCD XML documents submitted over HTTPS across diverse EHR systems. Handled via an auto-detecting channel with vendor-specific XSLT stylesheets that normalized varying document structures, consent locations, and patient identifiers.
• HL7 v2 Feeds: Transmitted via MLLP over TCP and HTTPS using custom Z-segments by regional exchanges and integration engine vendors. In the absence of an official IG mapping, consent was detected from display text, null fields caused malformed address and telecom arrays, and empty arrays left trailing punctuation. Handled through an explicit mapping matrix with client decisions per gap, consent detection mapped to LOINC codes, and frame decoders with manual parse fallbacks.
• Managed-Care File Flows: SFTP file drops with transmission metadata and outbound FHIR bundle deliveries for multiple regional payer and managed-care partners. Handled through interface engine channels, S3 bucket drops with decrypt-and-validate worker functions, and a dedicated multi-tenant role model.
• Reference Data: Code-set files (LOINC, SNOMED CT, ICD-10, CPT, HCPCS) and Implementation Guide packages managed by the state coordinating entity. Managed via expanded local reference files in pre-production, official files prior to production release, and header-selectable Implementation Guide versions.
• Sensitivity Markers: Security labels supplied through CSV columns, HTTP request headers (for C-CDA and HL7 v2), or pass-through FHIR tags. Handled by applying standardized security-label coding directly onto the target FHIR bundle.

Clinical Document Normalization Across Six EHR Sources

Variations across C-CDA exports required individual normalization strategies across six distinct EHR source categories:

• Source A (Enterprise EHR): Extension elements broke strict schema validation, member IDs, organization NPIs, and TINs were absent from payloads, consent documents were omitted entirely on negative responses, and race codes fell outside federal value sets. Handled by filtering unused sections before validation, shifting missing identifiers into HTTP request headers, and passing non-standard race values as detailed extensions.
• Source B (Small-Practice EHR): Contained no encompassing encounter structure, and consent was recorded strictly as a question-and-answer pair. Handled with a dedicated XSLT stylesheet in the shared intake channel, extracting MRNs from the first patient ID and ensuring consent-only files generated no spurious clinical observations.
• Source C (Cloud Ambulatory EHR): Lacked a screening grouper code, used prefixed MRNs, and relocated consent into social-history sections following an upstream vendor upgrade. Handled via optional screening-code request headers, layered consent parsing rules, and reprocessing thousands of historical documents with the state entity purging and replaying bundles.
• Source D (Health System Sources): Encounter statuses were stored in disparate source fields, and both NPI and TIN were required. Handled through an encounter status fallback sequence, populating organization resources with both identifiers, and reconciling test and production files.
• Source E (Ambulatory EHR): Sample payloads lacked screening and consent content because the vendor was transitioning to an alternative screening tool. Ingestion was deferred pending the vendor's product roadmap.
• Source F (Ambulatory EHR): Consent was held as a social-history question, the EHR vendor name was omitted from headers, and observations lacked effective timestamps. Handled by identifying the vendor through organizational OID, parsing consent from social history, and deriving observation identifiers from a four-step date fallback.

Logical Data Flow

1. 1 Senders (FHIR, CSV, C-CDA, HL7 v2, MCOs) Submit clinical, screening, and social-needs payloads over HTTPS, SFTP, or MLLP TCP into tenant-isolated gateway listeners.
2. 2 Ingestion API & Load Balancer Terminates traffic, verifies mTLS client certificates via forwarded headers, and applies path- and port-based routing driven by configuration.
3. 3 Raw Storage & Queue (S3 / SQS) Persists unmodified source payloads and metadata to S3 object storage and places tasks onto SQS queues prior to conversion.
4. 4 Validation & Transformation Layer Applies Frictionless CSV checks, vendor-specific XSLT transforms, HAPI FHIR validation, and JSON-path YAML reject rules.
5. 5 State Data Lake (Destination) Receives validated, conforming FHIR bundles via validate-and-forward API routes.
6. 6 Reject Triage Queue (Needs Attention) Records malformed or rejected records with root-cause error taxonomy in PostgreSQL, enabling operator review and bundle replay.

Key Architectural Decisions and Principles

• Validate No More Strictly Than the Implementation Guide: Rejected data only when explicitly mandated by the IG, avoiding unnecessary rejections; relaxed certain CSV rules while capturing missing attributes (such as dates of birth) in an operator triage view.
• Business Rules Governed as Data: Business validation rules were declared as YAML flags applied by a JSON-path evaluation engine, allowing rule adjustments without full code releases.
• Single Auto-Detecting C-CDA Channel: Maintained a unified intake channel with vendor XSLT stylesheets held in a centralized lookup table rather than maintaining separate channels per vendor.
• Contract Headers for Missing Facts: Shifted required identifiers (such as NPI and TIN) into HTTP request headers when senders were incapable of altering their legacy document exports.
• Routing Managed as Configuration: Migrated from per-port listeners to configuration-driven path routing on a single HTTPS listener, eliminating load-balancer listener and security-group quota exhaustion.
• mTLS in Passthrough Mode: The load balancer forwarded client certificates in HTTP headers for application-level verification against an S3 certificate authority bundle, avoiding termination bottlenecks.
• Store Raw Payloads in S3 Before Transformation: Original payloads were written to S3 and queued prior to conversion, ensuring raw source messages survived downstream crashes and remained available for replay.
• Decouple Ingestion Acknowledgment from Content Validation: Transport framing returned immediate framed acknowledgments (ACK/NACK), while validation details and error codes were logged asynchronously against unique interaction IDs.
• Historical Replay via Bundle-ID Prefixes: Enabled bundle replay over existing endpoints using configurable bundle-ID prefixes, requiring zero configuration changes on the sender side.
• Database Reader and Writer Separation: Read-heavy reporting and interactions grid queries were separated onto a dedicated read replica to protect write throughput on the ingestion database.
• Equal Sizing on Database Replicas: Adjusted database replica instance sizes to match the primary writer, preventing undersized reader instances from being promoted during automated failover events.
• Infrastructure Declared in AWS CDK: Container tasks, database failover alarms, and deployment pipelines were defined as code, eliminating manual console drift.
• Isolate Next-Generation Standards on Dedicated Branches: Evolving Implementation Guide releases were developed and tested on separate code branches, ensuring shared QA environments remained stable for active senders.
• Enterprise Identity Provider Migration: Migrated user authentication from basic GitHub OAuth to enterprise identity providers with nightly tenant synchronization.
• Database Migrations Ahead of Application Code: Enforced automated pre-deployment database schema updates before rolling out new application containers, preventing application boot failures.

Ingestion Business Rules

• Reject bundles with no Implementation Guide profile URL across all FHIR input.
• Derive bundle classification dynamically from profile URLs and state-supplied identification rules.
• Validate and reject invalid last-updated timestamps before attempting downstream forwarding.
• Detect consent strictly by standardized code rather than display text, accommodating absent consent, question-and-answer pairs, or social-history placements.
• Treat unknown consent as null rather than negative, ensuring missing values do not default to Deny or false.
• Carry missing metadata (member ID, organization NPI, TIN, facility, encounter type) in HTTP headers for C-CDA documents.
• Enforce a 64-character ceiling and a deterministic fallback sequence on observation and resource identifiers.
• Evaluate encounter status across a defined fallback sequence matching sender source fields.
• Transmit race values outside the federal standard as detailed extensions.
• Apply clinical data minimization by retaining only patient, organization, encounter, consent, and social-history sections, dropping allergies, medications, labs, and vitals.
• Enforce one location resource per C-CDA bundle, accept null flavor ASKU for gender, and exclude deceased timestamps from conversion.
• Validate CSV fields case-insensitively, enforce leap-year and month day limits, and return structured OperationOutcome responses on malformed rows.
• Require mandatory Z-segments on HL7 v2 feeds, returning generic NACKs while capturing detailed error diagnostics in internal logs.
• Append security-label codes to identify substance-use, mental health, and developmental disability records across all input types.

Documented Project Artifacts

• Endpoints and services for FHIR validation and forwarding, asynchronous CSV zip processing, C-CDA and HL7 v2 ingestion, failed-delivery querying, and bundle replay.
• Interface engine channels supporting SFTP, HTTPS, per-vendor XSLT lookups, MLLP HL7 v2 handling, and managed-care data delivery.
• Ingestion platform supporting routing configuration, mTLS validation, raw S3 payload storage, and SQS queue orchestration.
• Web interactions grid providing Needs Attention triage views, data-quality metrics for CSV, C-CDA, and HL7 v2, aggregate reporting, and an administrative diagnostic dashboard.
• AWS CDK infrastructure-as-code templates, CI/CD pipelines with prebuilt container images, CloudWatch alarms, automated release status announcements, and reader/writer database configurations.
• Remediation patches for external penetration test findings covering SQL injection on grid APIs, reflected XSS, missing HSTS policies, and insecure session cookies.
• Technical specifications covering CSV schemas, C-CDA mapping specifications, HL7 v2 custom segment matrices, negative-testing workflows, and deployment runbooks.

What the evidence supports — and what it does not

Relevance to Modern Healthcare Initiatives

This engagement demonstrates how health information exchanges (HIEs), regional data programs, and provider networks can ingest clinical and social-needs data across disparate external systems. Rather than assuming senders can immediately conform to ideal FHIR interfaces, the platform applies pre-validation checks, transparent reject categorization, and automated replay loops, ensuring downstream analytical stores receive clean, auditable records while keeping tenant boundaries intact.

Before embarking on an integration initiative of this scale, healthcare organizations should settle six operational questions:

• Who owns the schema rule set, and how frequently does it change?
• Can senders alter their output, or must missing metadata move into HTTP contract headers?
• What is the operational replay path when downstream endpoints experience outages, and who is authorized to trigger it?
• Can non-production environments remain completely free of identifiable patient data from day one?
• Who controls production promotion, and what is the release timeline from QA verification to live deployment?
• Are network configurations, failover thresholds, and alarms defined strictly in code?