The call started normally. A security audit, three weeks in, and the lead auditor was working through a data flow questionnaire. Standard stuff. Your CTO had answered forty questions in a row without hesitation.
Then she asked which third-party services processed customer data in the new AI-assisted onboarding feature.
There was a pause. Your CTO knew the LLM provider. He named it. She asked which product tier, and whether a Business Associate Agreement was in place. He said he would have to check. She asked whether the model’s data retention setting had been disabled. He said he would have to check that too. She asked whether the feature sent any PII before the document classification step or only after. He went quiet for a moment and then said he would pull the engineering team.
The call wrapped up two hours later with seven open items, a thirty-day remediation timeline, and a hold on the enterprise contract that had been three weeks from signing.
None of this was sabotage. The team had moved fast. They had shipped a working AI feature in six weeks. They had done everything right by the standards they had been applying. The problem was that they had never stopped to map which data was flowing to which AI endpoint, under what agreement, with what configuration.
The auditor found it because it is her job to find it. Most teams find out the same way.
The Coverage Gap Nobody Mapped
The compliance landscape for AI tools has not kept pace with how fast teams have adopted them. A HIPAA-compliant deployment of an AI product is a specific configuration, and it requires a specific product tier and a signed BAA. The default consumer or team tier of almost every major AI provider does not qualify.
The coverage maps look roughly like this. OpenAI offers BAAs for ChatGPT Enterprise, the Edu tier, and API customers on direct agreements. The standard ChatGPT Team account, which is what most engineering teams use to test and prototype, does not. Anthropic covers the API with a BAA available on request. The Claude consumer and Team products do not qualify, including the product powering some IDE-connected coding assistants. GitHub Copilot’s BAA applies only to Enterprise, which means the individual Business tier seats most developers use are outside coverage. Cursor does not currently offer a BAA at all.
This is not a problem with the providers. They have built compliant pathways. The problem is that most teams built their features on the prototyping tier, shipped when the feature was ready, and never revisited the compliance tier question after the prototype became production.
The Three Things the Audit Actually Finds
Not every compliance failure looks the same. The teams that have gone through this find three distinct problems, usually in combination.
The first is the tier problem described above. The wrong product tier is the most obvious failure mode and the quickest to remediate in theory, but slower in practice than it sounds. Upgrading from a team plan to an enterprise agreement involves procurement, contract review, and legal sign-off. That process takes weeks, and in the meantime, the feature is either running on a non-compliant setup or offline. Neither is comfortable.
The second problem is configuration. Even teams that are on the correct tier with a signed BAA often have not changed the default settings that affect compliance. Data retention is the most common gap. Most AI APIs log request and response data by default. Turning that off is a one-line configuration change, but it is opt-out, not opt-in, and if nobody made the choice explicitly, the data is being retained. Prompt logging, model fine-tuning eligibility, audit trail formats: each of these has a default setting that the provider chose for their own operational reasons, not yours. An enterprise API agreement gives you the right to change them. It does not change them for you.
The third problem is the most underrated. It is not about your vendor at all. It is in your customer contracts.
Enterprise customers in FinTech and HealthTech routinely include data processing restrictions in their service agreements that prohibit sending their data to subprocessors who are not explicitly listed and approved. Your enterprise contract says what you can do with their data. It was written before your AI feature existed. It almost certainly does not list your LLM provider as an approved subprocessor, because your legal team drafted it before the AI feature was scoped. The feature you shipped is technically in breach of that customer’s contract whether or not you have a valid BAA with your AI provider.
This is the one that catches teams hardest, because the fix is not a configuration change or a tier upgrade. It requires going back to the customer, disclosing what changed, and getting an amendment or addendum approved. That conversation is uncomfortable. It is also necessary, and it is easier to have proactively than after an incident.
What Teams Do When They Find Out
The reactions fall into a predictable set. Some teams freeze the AI feature while remediation is sorted out, which creates its own problems: the feature is suddenly unavailable, the team that built it is pulled off current work to fix it, and trust with the enterprise customer is dented regardless of the technical outcome.
Some teams rush to anonymize or strip PII before the data reaches the AI provider. This works in cases where the regulated data is separable from the input the AI actually needs. A document classifier that only needs the document type and general category can operate on an anonymized version. A contract analysis feature that needs to reason about specific dates, parties, and clause language cannot, because the regulated information is the signal. Anonymization solves a narrow subset of these cases.
Some teams go further and move to on-premises or private cloud deployments of smaller open-weight models. This is the right architectural choice for some applications, particularly in healthcare where PHI is consistently embedded in the reasoning context. It is not a quick fix. Deploying and maintaining a private model infrastructure is a months-long engineering investment, and the models available for private deployment are not yet equivalent in capability to the frontier models on managed APIs. Teams that make this choice are usually doing it because they knew the compliance requirement from the start, not as a retroactive response to an audit.
The pattern that works best, when it is available, is the one that requires the least retrofit. A FinTech or HealthTech team that maps its data classification requirements before building an AI feature can make a correct architecture choice upfront, and usually finds that it costs around twenty to thirty percent more to build but nothing extra to maintain and nothing to remediate later. The same team that skips that step and builds on a prototyping tier typically faces a retroactive cost that is two to three times the original build, plus the enterprise contract friction.
The Architecture Decision You Need to Make Before You Write a Line of Code
The compliance question for a regulated-industry AI feature is not “are we HIPAA compliant?” It is a sequence of more specific questions.
What type of data does the feature process, and what regulations govern it? Financial transaction data, health records, insurance identifiers, and payment card data each have distinct rules. A feature processing patient symptom descriptions operates under different constraints than one analyzing lab result trends, even though both involve healthcare data. Knowing which category your data falls into before you build determines which vendor tiers are available to you, and which are not.
Which specific data elements reach the AI provider, and when? A well-designed AI-powered development process maps the data flow before the first sprint planning session. Many AI features can be structured so that the regulated data stays in your infrastructure and only non-regulated context reaches the external model. This requires a deliberate architecture decision, not a retrofit.
What do your customer contracts say, and when were they last updated? If you have enterprise customers in regulated industries, your legal team needs to review your standard service agreement to ensure subprocessor provisions cover AI tools and your standard DPA includes the vendors you are using. This is a legal and procurement task, but engineering and product need to flag it early, because updating customer contracts takes time that does not show up in sprint estimates.
What does your vendor agreement actually cover? Signing a BAA is a starting point. Reading it closely enough to understand which of your vendor’s products it covers, which data retention settings it obligates you to configure, and what your obligations are in the event of an incident is the step most teams skip.
This is exactly the kind of architecture work that is easy to describe as compliance overhead and easy to deprioritize when a feature is three weeks from launch. It is also exactly the kind of work that application modernisation and AI consulting engagements are well positioned to tackle upfront, precisely because the people doing the work have done the mapping before and know where the gaps tend to be.
It Is Fixable. It Just Is Not Free.
The CTO on that audit call fixed everything. It took six weeks, a legal negotiation with the enterprise customer, a vendor tier upgrade, a configuration audit, and two sprints of engineering time that was not on the roadmap. The enterprise contract signed seven weeks late.
The story has a good ending. Most teams who go through this version of it come out with a cleaner, more defensible AI architecture than they had before. The audit pressure that feels like a disruption usually forces a set of decisions that should have been made at the design phase.
The better version is making those decisions before the auditor asks. Not because compliance paperwork is interesting but because the architectural conversation is the same one either way. If you are building AI features on customer data in a regulated industry, the question of which data flows where and under what agreement is not a compliance checkbox. It is an infrastructure decision that belongs in your design review, alongside your data models and your service boundaries.
The auditor is going to ask which tier you are on, whether the BAA is signed, whether data retention is disabled, and whether your customer contracts permit it. Getting those answers right before the call is not just about passing the audit. It is about building something you can actually stand behind.