How to Audit Your AI Vendors: A Practical Guide to Third-Party Risk

TL;DR

• Most companies have more AI vendors than they realize, and you're liable for what those vendors do.
• A patchwork of state laws makes vendor oversight non-negotiable. California, Illinois, Colorado, NYC, and Texas all hold employers responsible for third-party AI tools.
• A five-step audit gets you most of the way there: inventory, documentation, outcome data, technical review, governance.
• Continuous monitoring is the new standard. One-time vendor reviews don't keep pace with model and regulatory change.

Most companies have far more AI vendors than they realize, embedded in everything from resume screening to expense automation. Under the new wave of state AI laws, you're liable for what those vendors' AI does, even when you didn't build it. "The AI did it" isn't a defense, and "the vendor did it" generally isn't either.

If you're in Compliance, Legal, HR, or People Ops, here's a practical guide to auditing what you already have. (Want a shortcut to an AI Vendor Audit? Grab our checklist now.)

Terminology to Know

Before auditing, get fluent in the key terms:

How to Audit Your AI Vendors

Step 1: Inventory and scope

List every AI tool in use across all departments (procurement records will miss the long tail). For each tool, document what it does, who uses it, what data it processes, and whether it influences a consequential decision. Priority targets: any tool touching consequential decisions.

Step 2: Documentation requests

Request from each priority vendor:

• Most recent bias audit
• Impact assessment or risk assessment
• Training data sources and update frequency
• Model performance metrics by demographic group
• SOC 2 or equivalent security audit
• Data processing agreement terms

If a vendor can't or won't provide this, that's a finding.

Step 3: Outcome data review

Run your own data through the four-fifths rule: calculate selection rates by demographic group at each stage where the AI is involved, and flag any group whose rate falls below 80% of the highest-performing group. Your vendor's audit is on their aggregate data; yours is on yours.

Step 4: Technical review

For each tool, document the vendor's answers to:

• Is your data used to train the model?
• How often is the model retrained, and on what data?
• Can decisions be explained to an affected individual?
• What's the process for detecting and correcting drift?

If the vendor can't answer clearly, that's another finding.

Step 5: Governance and contract review

Pull the contract. Check for indemnification clauses (Colorado's new AI Act voids certain employer-unfavorable terms), breach notification timing, right to audit, data ownership and training rights, and termination rights tied to risk-profile changes.

For global teams: If your AI touches people outside the US, contract and governance review expands significantly. The EU AI Act's high-risk obligations take effect August 2, 2026, covering most workplace AI (recruitment, performance management, worker monitoring) with extraterritorial reach and penalties up to €15M or 3% of global revenue for high-risk system non-compliance. The UK uses principles-based oversight across existing regulators. South Korea's Basic AI Act entered force January 22, 2026, also with extraterritorial reach.

Audit Rubric

Want a shorter version for your audits? Download the AI Vendor Audit Checklist.

What to Do When You Find Risk

For each tool flagged as "Action Needed" or with multiple "Gap" scores:

1. Document the finding. Date, scope, evidence, initial assessment.
2. Request remediation. Set a 30-day deadline.
3. Add human review. Insert a checkpoint in any decision flow the tool affects.
4. Evaluate alternatives. Identify competitors with stronger compliance posture.
5. Update contracts. Negotiate indemnification, audit rights, and breach notification at next renewal.
6. Pull the tool if needed. If the vendor won't remediate and risk is material, exit.

Continuous monitoring is the new standard. Ethena's Third-Party Risk Agent screens your vendors and partners for ethics and compliance exposures before they become your liability: continuous monitoring, not a one-time checkbox. Talk to our team about how we can get compliance agents working for you.

Frequently Asked Questions

Q: What if our vendor won't share their bias audit or model documentation?

A: That's a finding. Document the refusal in writing. State AI laws hold the employer-deployer responsible regardless of vendor disclosure. If you can't get the documentation to demonstrate compliance, you can't defensibly use the tool.

Q: What's the difference between a bias audit and an impact assessment?

A: A bias audit is backward-looking, analyzing an AI tool's outcomes across demographic groups. An impact assessment is forward-looking, documenting the tool's purpose, risks, and mitigations before deployment. California's CPPA rules and Colorado's new AI Act both require impact assessments.

For more on the state-by-state requirements driving vendor liability, see our companion post on AI in Hiring: 5 Employment Laws Every HR Team Should Know in 2026.

How Ethena Can Help

Ethena's training on AI in the Workplace and AI in Hiring helps your team use AI responsibly and recognize vendor risk. For ongoing vendor monitoring, our Third-Party Risk Agent does the work continuously. Let's talk.

Disclaimer: None of the content in this article constitutes legal advice, nor does it contain every detail or requirement of the applicable laws. It is provided solely for informational purposes and is not intended to be relied upon as a standalone resource. If you have questions about these laws or their implications for your organization, please consult your legal counsel.