AI Regulation in Healthcare: Federal & State Requirements

Key Requirements

• Market authorization required for AI/ML devices meeting the SaMD definition — either 510(k) premarket notification (approximately 97% of AI/ML devices), De Novo classification, or PMA (highest risk). Cumulative cleared/authorized AI/ML devices: approximately 1,200 as of 2025; 295 cleared in 2025 alone.
• Labeling requirements: Draft guidance (January 2025) specifies required labeling elements for AI/ML-enabled devices — including algorithm description, training data characteristics, validation performance data, known limitations, and human oversight requirements.
• Predetermined Change Control Plan (PCCP): Required for adaptive algorithms that update post-deployment. The PCCP defines in advance how the algorithm may change within pre-specified safety parameters without requiring a new submission. 30 devices (10% of 2025 clearances) included authorized PCCPs. Organizations deploying adaptive algorithms without an authorized PCCP may be making unapproved device modifications.
• Post-market surveillance: MDR (Medical Device Reporting) obligations apply to AI/ML devices — malfunctions and adverse events including AI hallucinations or performance failures must be reported.
• Average review time: 150 days for AI/ML device clearance. Documentation quality directly affects timeline.

Impact on Health Plans

Health plans that license AI-enabled clinical decision support tools from vendors bear secondary exposure — if the vendor's AI tool requires FDA clearance and lacks it, the health plan's use of that tool in clinical workflows creates regulatory exposure. Vendor AI governance due diligence should include verification of FDA authorization status for any AI used in clinical decision-making.

FY2026 User Fees

• PMA (full application, standard): $531,163
• PMA (small business): $113,750
• 510(k): significantly lower than PMA; specific amounts published in annual Federal Register notice

Key Requirements — Section (b)(11) Source Attributes

Certified health IT that includes predictive Decision Support Interventions must provide clinical end-users with source attribute information for each predictive DSI. Required source attributes include:

• Identity of the intervention developer
• Funding sources used to develop the intervention
• Intervention description and intended use
• Training data characteristics: demographic breakdown, exclusion criteria, known limitations
• Intervention output and the rationale for how the output was determined
• Information about how the intervention performed in testing environments, including information on fairness, bias, and equity

Impact on Health Systems

Health systems using certified EHRs with predictive clinical decision support features must verify that their EHR vendor has configured source attribute documentation for all predictive DSI features. If the vendor has not implemented HTI-1 transparency requirements, the health system's certified EHR is out of compliance — which can affect Meaningful Use attestation and CMS incentive payment eligibility.

Compliance Action Required

• Inventory all predictive DSIs deployed in certified EHR environment
• Verify vendor has implemented Section (b)(11) source attribute configuration
• Ensure source attributes are accessible to clinical end-users at point of care
• Document compliance for audit purposes

Key Requirements

• Prior authorization transparency: CMS Interoperability and Prior Authorization rules (CMS-0057-F) require health plans to provide specific reasons for prior authorization denials — AI-generated denial recommendations must be traceable, auditable, and explainable to members and providers.
• RADV audit exposure: CMS is deploying AI-driven analytics to complete Payment Year 2018–2024 RADV audit backlog. Organizations that used AI-assisted coding that inflated diagnosis code capture face significant recoupment exposure. AI-assisted risk adjustment coding programs must have documented governance and human review protocols.
• Health equity obligations: CMS contracts with Medicare Advantage and Medicaid managed care organizations include health equity requirements. AI-assisted utilization management tools must be audited for differential impacts on protected populations.

Impact on Health Plans

Medicare Advantage plans using AI in coverage or prior authorization workflows face the highest CMS scrutiny. Documented human oversight of AI coverage recommendations, audit trails for AI-influenced decisions, and bias auditing for differential denial rates across demographic groups are the core compliance requirements. Colorado HB 1139 (state law) prohibits AI-only coverage denials — health plans operating in Colorado must ensure human clinical review before any AI-generated denial recommendation becomes a final coverage decision.

Key Requirements for AI

• Minimum Necessary Standard: PHI used for AI model training must be limited to the minimum necessary to accomplish the training purpose. Bulk PHI exports for vendor model training without minimum necessary analysis are potentially non-compliant.
• Business Associate Agreements: Must specifically address: what PHI the AI vendor may use for training purposes, how it must be de-identified or limited, subprocessor obligations for AI infrastructure providers, and breach notification protocols for AI training environments.
• De-identification: AI training datasets must satisfy Safe Harbor or Expert Determination de-identification standards. "Anonymized" datasets that can be re-identified through model outputs or inference attacks do not qualify as de-identified under HIPAA.
• Breach notification: AI-related incidents — including data poisoning attacks, unauthorized model extraction, or AI hallucinations that result in unauthorized PHI disclosure — trigger HIPAA breach notification obligations.
• Security Rule: AI systems that process, store, or transmit PHI are subject to HIPAA Security Rule administrative, physical, and technical safeguard requirements, including Software Bill of Materials management for AI/ML components.

Framework Structure

The NIST AI RMF 1.0 organizes AI risk management across four core functions:

• Govern: Establish organizational culture, policies, and accountability structures for AI risk management
• Map: Categorize AI systems, identify context and risks associated with each deployment
• Measure: Analyze and assess AI risks using appropriate metrics and methods
• Manage: Prioritize and treat AI risks; document residual risks

Healthcare Adoption Gap

84% of healthcare organizations have established AI governance committees, but only 12% have implemented a formal AI governance framework such as NIST AI RMF (Censinet/CHIME Foundation, 2025). This gap — between committee existence and framework implementation — is where regulatory exposure lives. A committee that meets but produces no NIST-mapped documentation does not satisfy regulatory requirements that reference the AI RMF.

Medical AI Classification as High-Risk

The EU AI Act classifies AI systems used as safety components in medical devices or as medical devices themselves — under EU MDR Class IIa/IIb/III or IVDR Class A-D — as high-risk AI. High-risk AI triggers the full compliance framework:

• Risk management system: Documented AI risk management processes aligned with ISO 31000 and ISO 14971 (medical device risk management)
• Data governance: Training, validation, and testing data must meet quality criteria; data governance documentation required
• Technical documentation: Comprehensive technical file including system description, design specifications, performance metrics, and validation data
• Transparency and provision of information: Instructions for use disclosing capabilities, limitations, and human oversight requirements
• Human oversight measures: Documented mechanisms enabling human monitoring and intervention
• Accuracy, robustness, and cybersecurity: Validation against applicable standards; cybersecurity requirements aligned with EU Medical Device Regulation
• Conformity assessment: Third-party conformity assessment by a notified body or self-assessment depending on classification
• EU database registration: Registration in the EU AI Act database before placing the system on the EU market
• Quality management system: ISO 13485-aligned QMS required for medical AI

Extraterritorial Application for U.S. Organizations

The EU AI Act applies extraterritorially in a manner comparable to GDPR. U.S. healthcare organizations face EU AI Act obligations if: (1) they operate healthcare facilities or services in EU member states; (2) they license health IT or AI-enabled medical device software to European healthcare providers, health plans, or patients; or (3) their AI systems produce outputs that affect EU residents, regardless of where the AI system is hosted. U.S. health IT vendors selling EHR or clinical decision support software to European hospital customers — even without a European corporate presence — are subject to the EU AI Act requirements for their AI-enabled features.

Penalties

Violations of EU AI Act requirements for high-risk AI systems: up to €30 million or 6% of global annual turnover, whichever is higher. For prohibited AI practices: up to €35 million or 7% of global annual turnover.