Private AI for healthcare: what actually needs to stay inside your VPC?
“Private AI” is an easy phrase to say and a hard architecture to design. Healthcare teams do not need every pixel, dashboard, and support workflow locked inside the same private network. They do need a clear answer to a more important question: where can protected health information, clinical context, model inputs, outputs, retrieval data, and audit evidence safely exist?
The practical goal is not isolation for its own sake. The goal is control. A private AI deployment should keep the components that create, receive, maintain, transmit, or materially expose PHI inside a hospital-controlled trust boundary, while allowing lower-risk product surfaces to stay operationally manageable.
Start with the trust boundary, not the vendor diagram.
In cloud architecture, a VPC is a useful boundary because it gives the healthcare organization control over private networking, routing, security groups, subnets, peering, endpoint policies, logging, and connectivity to internal systems. But a VPC is not magic. Putting software inside a private subnet does not automatically make the workflow safe.
A better design question is: which systems need to sit inside the same governed environment as the data they operate on? For healthcare AI, that usually means the parts of the stack that touch PHI or can reconstruct it.
Patient records and source documents
EHR extracts, uploaded PDFs, referral packets, discharge materials, labs, notes, and other patient-specific documents should remain in the customer-controlled environment.
Vector indexes and embeddings
Embeddings and retrieval indexes can reveal sensitive source content or patient context. Treat them as governed clinical data, not harmless metadata.
Prompt assembly and model inference
The final prompt often contains PHI, retrieved passages, clinician intent, and patient-specific instructions. In high-PHI workflows, inference belongs inside the private boundary.
Audit logs and generated outputs
Outputs, edits, exports, access events, retrieved sources, and administrative actions can all contain PHI or compliance evidence.
Marketing site and public docs
Public product pages, documentation, status pages, and non-sensitive education content usually do not need to live in the clinical VPC.
De-identified operations analytics
Some telemetry can live outside the VPC if it is stripped of PHI, contractually controlled, and reviewed by security and compliance teams.
The VPC decision matrix.
Clinical source data
Patient records, internal policies, local protocols, and uploaded documents should stay private because they are the raw material for clinical work.
Only when truly public, de-identified, or approved for external processing under the organization’s policy.
Retrieval layer
Search indexes, embeddings, chunk stores, and rerankers should sit beside the governed corpus and inherit the same permissions.
Only for public medical references or isolated, non-PHI knowledge bases where leakage would not expose patients or operations.
Model runtime
Inference should stay private when prompts include PHI, retrieved chart context, internal policies, or clinical documentation drafts.
For low-risk education, public content summarization, or de-identified pilots with a signed BAA and clear retention controls.
Application UI
The authenticated clinical workspace often belongs near the data plane so sessions, attachments, and generated outputs remain governed.
Static marketing pages, public docs, and non-PHI admin surfaces may be hosted elsewhere.
Monitoring and support
Logs containing prompts, outputs, document IDs, user actions, or patient references should remain private.
Aggregated health metrics, deployment status, and non-sensitive performance telemetry may leave the VPC after PHI review.
Five principles for private healthcare AI.
Put PHI-handling components under customer control.
If a service creates, receives, maintains, transmits, stores, retrieves, logs, or transforms PHI, assume it belongs inside the customer-controlled environment unless security and legal teams decide otherwise.
Treat derived data as sensitive.
Embeddings, summaries, generated drafts, extracted template fields, cache entries, and support traces may not look like raw records, but they can still reveal patient information or institutional context.
Use private connectivity for clinical systems.
EHR integrations, FHIR servers, object storage, data warehouses, identity providers, and key management services should connect through private routes where possible, not broad public endpoints.
Design for least privilege, not perimeter comfort.
A private VPC should still use role-based access, scoped service identities, network segmentation, encryption, key separation, and per-request authorization. Internal does not mean trusted.
Keep evidence with the workflow.
Healthcare AI needs reviewability: who asked, which sources were retrieved, what the model generated, what the clinician changed, and what was exported. Audit evidence should follow the same governance boundary.
What does not automatically need to be inside the VPC.
A strong private AI architecture can still use external services when the risk is understood and controlled. The mistake is not using a vendor. The mistake is letting PHI leak into places the organization cannot govern, inspect, or contractually control.
- Public web pages, product documentation, and release notes.
- Generic product analytics that never include prompts, outputs, user-entered text, document names, patient identifiers, or clinical content.
- Status monitoring that reports service health without exposing tenant, patient, or workflow detail.
- External support workflows that use redacted tickets, synthetic examples, or customer-approved diagnostic bundles.
- Public medical references, as long as retrieval does not combine them with patient-specific context outside the boundary.
The BAA is necessary, but it is not the architecture.
HHS guidance makes clear that a cloud service provider that creates, receives, maintains, or transmits ePHI for a covered entity or business associate is generally a business associate and needs a HIPAA-compliant agreement. That matters. But a BAA does not answer the technical questions by itself.
The architecture still has to define where data is stored, where inference runs, how keys are managed, who can access logs, whether support can view customer content, how backups work, how data is deleted, and what happens when a model or retrieval system is updated.
A useful test: if a security reviewer asks, “Where can PHI go?” the answer should be a diagram, not a promise.
Questions to ask any private AI vendor.
- Can the model runtime, vector database, document store, and audit logs run inside our cloud account or dedicated VPC?
- Which components ever receive prompts, retrieved passages, generated outputs, document names, user identifiers, or patient identifiers?
- Are embeddings, caches, traces, support bundles, and evaluation datasets treated as sensitive derived data?
- Can we use our own identity provider, key management, logging, network controls, and security monitoring?
- Can vendor staff access customer content, and if so, how is that approved, logged, time-limited, and reviewed?
- What data leaves the VPC by default, and can every outbound path be disabled, inspected, or allowlisted?
How Council approaches private AI.
Council is built for healthcare organizations that want AI without giving up control of patient data. For some teams, that means running Council 1 or the CouncilAI Platform in a customer-controlled cloud environment. For others, it means a staged path: start with a constrained pilot, then move PHI-heavy workflows into a private deployment as governance matures.
The design principle stays the same: keep clinical context, retrieval, inference, generated outputs, and audit evidence where the healthcare organization can govern them. Private AI should make security review more concrete, not more mysterious.
The private AI question is not “cloud or no cloud?”
It is: who controls the systems that see clinical data, who can prove what happened, and who decides where the next byte of PHI is allowed to go?
Further reading.
Useful primary references include HHS guidance on HIPAA and cloud computing, HHS guidance on business associates, and NIST Special Publication 800-207 on zero trust architecture.
Bring AI closer to the data you govern.
Council helps healthcare teams deploy clinical AI with controlled infrastructure, knowledge retrieval, document workflows, and audit-ready evidence.