When High-Risk AI Systems Applies and What to Do Next

High-risk AI systems applies when a SaaS team builds, sells, embeds, configures, or uses an AI system that falls into one of the EU AI Act's high-risk routes. The practical question is not whether the company uses AI somewhere. The practical question is whether a specific system, with a specific intended purpose, sits in a regulated product context or in one of the sensitive use cases listed in Annex III.

For SaaS teams, that answer often depends on details that live outside legal: product purpose, customer configuration, affected people, data categories, vendor role, human review, market placement, and how the output is used. A feature that looks ordinary in one workflow can become sensitive in another.

The right next step is a repeatable classification workflow. The team should identify the system, screen the use case, assign ownership, document the decision, and route likely high-risk systems into deeper controls before launch or customer commitment.

The two main routes

The first route is the regulated-product route. An AI system can be high-risk when it is intended to be used as a safety component of a product, or is itself a product, covered by Union harmonisation legislation listed in Annex I, and that product or system requires third-party conformity assessment. This route matters most for SaaS connected to medical devices, machinery, transport, aviation, radio equipment, toys, industrial systems, robotics, or other regulated product environments.

Many ordinary B2B SaaS products will not enter this route. But teams serving healthcare, mobility, industrial automation, device, infrastructure, or safety-critical markets should not dismiss it. If the AI output supports safety-relevant control, diagnostics, monitoring, triage, certification, or product behavior, the review needs product regulatory expertise.

The second route is Annex III. This is often the more visible route for SaaS. Annex III covers listed areas such as biometrics, critical infrastructure, education and vocational training, employment and worker management, access to essential private and public services, law enforcement, migration and border control, administration of justice, and democratic processes.

The analysis depends on intended purpose and concrete use. A model used to summarize meeting notes may be low risk. A similar model used to rank candidates, evaluate workers, support student assessment, influence credit eligibility, or route people into essential services can require a much deeper review.

When SaaS teams should trigger a review

Trigger high-risk review whenever an AI system can affect people in a consequential context. The team does not need a final legal answer before collecting facts. The first job is to notice that the question exists.

Common triggers include a new AI feature, a new model or vendor, a new intended purpose, a new customer configuration, use in employment or HR, use in education, use in healthcare or essential services, use in credit or insurance, biometric processing, automated ranking or eligibility support, reduced human review, expansion into the EU market, or a customer question about AI Act exposure.

Customer configuration deserves special attention. A horizontal SaaS platform may not be high-risk in its default use, but it can become sensitive when customers use it to assess workers, rank applicants, score eligibility, route students, detect identity, or support public-sector decisions. Product teams should define configuration triggers that require review before enablement.

When it may not apply

Not every AI-enabled SaaS feature is high-risk. An internal drafting assistant, code helper, product analytics assistant, support-note summarizer, or knowledge search tool may be outside the high-risk route if it does not operate in an Annex III context, is not a safety component of a regulated product, and does not support consequential decisions about people.

That does not mean the feature is compliance-free. It may still need AI inventory entry, privacy review, security review, vendor assessment, prohibited-practice screening, customer disclosure, or ordinary AI governance controls.

The point is proportional routing. Low-risk systems should move quickly with basic evidence. Possible high-risk systems should receive deeper review before launch. Ambiguous systems should have a named reviewer and deadline instead of floating in a risk register.

What to do first

Start with a short intake that product and engineering can complete without legal translation. Capture system name, owner, business purpose, user journey, affected people, geography, data categories, model or vendor, AI Act role hypothesis, output, human review, customer configuration, and whether the use case touches regulated products or Annex III areas.

Then assign a route. Clear no-high-risk cases proceed to ordinary AI governance, privacy, security, and vendor review. Clear high-risk candidates move into deeper legal and compliance assessment. Uncertain cases go to a named reviewer with a deadline and a minimum evidence package.

For likely high-risk systems, expand the record. Capture risk management, data governance, technical documentation, logging, transparency, instructions for use, human oversight, accuracy and robustness testing, cybersecurity measures, monitoring, incident escalation, vendor evidence, conformity-assessment route, and customer documentation.

Evidence to keep

Good evidence should answer four questions: what did the team review, why did it classify the system this way, what controls followed, and when will the decision be reopened?

Keep the AI inventory entry, intake form, product or vendor description, data-flow note, affected-person analysis, intended-purpose statement, Annex I or Annex III screening, role analysis, classification conclusion, reviewer, date, rationale, launch decision, controls triggered, and re-review trigger.

If the system is built internally, keep architecture notes, model documentation, testing evidence, logging design, human oversight design, and monitoring plan. If a vendor is involved, keep the vendor's AI documentation, instructions for use, contractual commitments, security questionnaire answers, audit or certification materials, and any statement about role allocation.

Store evidence where teams already work. Product tickets, architecture records, privacy reviews, vendor records, release checklists, and trust-center folders are usually stronger than a standalone spreadsheet.

Common SaaS scenarios

An AI support assistant that summarizes internal tickets for agents may be outside the high-risk route if staff review the output and the system does not rank, score, or decide access for people. The team should still document vendor terms, data flow, privacy impact, and customer commitments.

An HR SaaS feature that filters applications, ranks candidates, evaluates worker performance, or supports termination decisions is different. Employment and worker management are Annex III areas. The team should trigger deeper review, define human oversight, test data quality and bias risk, write customer instructions, and keep evidence of monitoring.

A healthcare workflow that helps prioritize treatment, diagnostics, triage, or medical-device functionality may raise both Annex III and regulated-product questions. The team should identify whether the system is a safety component, whether sector rules require third-party conformity assessment, and which obligations belong to provider, deployer, manufacturer, customer, or vendor.

A workflow platform that allows customers to configure AI automations needs guardrails. The vendor may need prohibited-use rules, admin controls, customer-use questionnaires, support escalation, logging, and contractual limits so sensitive configurations do not bypass review.

Connect it to the operating model

Product should own purpose, journey, configuration, and launch facts. Engineering should own architecture, data flow, model integration, logs, and technical controls. Security should own vendor and access review. Privacy should own data protection impact. Legal and compliance should own AI Act interpretation, role analysis, evidence standard, and approval conditions.

Tie the workflow to product intake, architecture review, vendor onboarding, privacy review, security review, sales commitments, customer configuration, and release approval. That keeps high-risk review from becoming a separate legal exercise.

This also connects to broader AI governance expectations for SaaS vendors, internal AI-tool review, the EU AI Act overview for SaaS providers, and the controls buyers increasingly ask about for AI-enabled SaaS products.

FAQ

What is the practical purpose of high-risk AI systems?

The practical purpose is to identify AI systems that need a stricter governance path, stronger evidence, lifecycle controls, and clearer ownership before they are placed on the market, put into service, or used in sensitive workflows.

When does high-risk AI systems apply to SaaS teams?

It may apply when a SaaS team builds, sells, embeds, configures, or uses AI as a safety component of a regulated product, as a regulated product, or for an Annex III use case such as employment, education, essential services, biometrics, law enforcement, migration, justice, or democratic processes.

What should teams document or change first?

Start with an AI inventory, high-risk intake screen, role analysis, intended-purpose statement, affected-person analysis, classification decision, named owner, evidence location, launch gate, and re-review trigger.

Is every AI-enabled SaaS product high risk?

No. Many AI-enabled SaaS features will not be high-risk. The team still needs to document why the use case does or does not enter the high-risk route, especially when people are ranked, scored, filtered, monitored, or given access to important services.

What happens if the answer is unclear?

Route the case to a named reviewer, collect the missing facts, set a decision deadline, and block only the sensitive launch path or customer configuration that needs review.

Sources

• Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence.
• European Commission guidance for providers and deployers of high-risk AI systems.
• European Commission AI Act FAQ.
• AI Act Service Desk guidance on Annex III high-risk AI systems.