Generative AI risk assessments go wrong in a predictable direction. They try to assess the model: its training data, its benchmark scores, its safety card. That work is mostly done by the model provider, it is the same for every customer, and it rarely changes the decision.

What changes the decision is the use. The same model summarizing internal meeting notes and drafting replies to customers' billing disputes carries completely different risk, because different people rely on the output, different data goes in, and the cost of an error is different.

This brief is for the cases the intake process routes to full assessment: regulated or restricted data, actions without a person in the loop, decisions about people, or customer-facing output at scale.

The public references, briefly

Three public documents are worth using and none of them is a checklist you can complete mechanically.

  • NIST AI RMF 1.0, released in January 2023, is voluntary and is being revised under the White House AI Action Plan. Its Map function is where use-case context lives: intended purpose, settings, and the likelihood and magnitude of impacts.
  • NIST AI 600-1, the Generative AI Profile, published in July 2024, defines twelve risks novel to or exacerbated by generative AI. It is the most useful scoping list available for free.
  • ISO/IEC 42001:2023 specifies requirements for an AI management system and applies to any organization providing or using AI. It is where a repeatable assessment process gets audited, not where a single assessment is designed.

For a section-by-section explanation of the AI RMF, see NIST AI RMF in practice. This brief does not repeat it.

The four questions

1. Who relies on the output, and how?

Name the reader. An employee who reviews every draft is a very different control from a customer who receives the output directly. Write down whether a person reviews before use, whether that review is realistic at the expected volume, and whether the reader knows the output is AI-generated.

2. What data goes in?

The highest data class in prompts, retrieved documents, and connected systems. Include what the system can retrieve, not just what users paste. Then the vendor terms: retention, whether inputs are used for training, where processing happens, and what happens to data at contract end.

3. What can it do without a person?

Text only, or actions? Can it send, change, approve, pay, or grant? Every action needs a credential, an owner, and a revocation path, which makes it an identity question as much as an AI one.

4. What happens when it is wrong?

Describe the failure in concrete terms: a wrong figure sent to a customer, a confidential document summarized into a shared channel, an approval granted that should not have been. Then estimate how often a reviewer would catch it and how fast you could undo it.

Using the twelve risks as a scoping list

NIST AI 600-1 names twelve risks. Read each against the use case and mark it in scope, out of scope, or unknown. Expect only a handful to be in scope for any one use case.

NIST AI 600-1 riskTypical question for an enterprise use case
CBRN Information or CapabilitiesRarely in scope outside research settings
ConfabulationWhat breaks if the output is confidently wrong?
Dangerous, Violent, or Hateful ContentCan users or inputs steer it to produce this?
Data PrivacyDoes personal data go in, or could it come out?
Environmental ImpactsUsually a procurement note, not a blocker
Harmful Bias and HomogenizationDoes it shape decisions or communications about people?
Human-AI ConfigurationWill reviewers over-trust it, or ignore it?
Information IntegrityCould output be mistaken for verified fact?
Information SecurityPrompt injection, data exfiltration, insecure integrations
Intellectual PropertyLicensed or third-party content in inputs or outputs
Obscene, Degrading, and/or Abusive ContentPublic-facing generation, image tools
Value Chain and Component IntegrationWhich third-party models, plugins, and data sources sit underneath?

Use the list to make sure nothing obvious was skipped. Do not turn it into a weighted score: a total across twelve unrelated risks hides the one that matters.

Why the obvious answer is wrong

The obvious answer is a scored questionnaire that produces a number and an automatic decision. It feels objective. In practice the weights are invented, the score is negotiated, and the decision that matters, which condition makes this acceptable, never gets written down.

The second obvious answer is to defer to the vendor's assurances: a certification, a trust page, a model card. Those describe the vendor's controls. They say nothing about whether your reviewers will read every output, or what your connected systems allow the tool to do.

The discriminating variables

Is a person actually in the loop?

The test: estimate outputs per reviewer per day. If one person is expected to check hundreds of drafts, the review is nominal and the use case is effectively autonomous.

Is the data class above what the vendor terms allow?

The test: compare the highest data class with the contract. If inputs may be retained or used for training and the data is confidential or above, the answer is no until the terms change.

Can it act?

The test: list every system it can write to. Any write access means an owner, a scoped credential, logging, and a tested way to switch it off.

Could it fall under a regulated category?

The test: does it shape decisions about employment, credit, education, access to services, or similar? If so, involve counsel. Under the EU AI Act, high-risk obligations for those Annex III use cases apply from 2 December 2027 after the Omnibus amendment, and classification depends on facts a security assessment cannot settle alone.

Decision table

FindingDecisionTypical conditions
Person reviews every output, internal data, no actionsApproveLogging on, annual review
Confidential data, acceptable vendor termsApprove with conditionsEnterprise account only, retention off, access scoped
Customer-facing output at volumeApprove with conditionsDisclosure to users, sampling review, complaint path
Write access to business systemsApprove with conditionsNamed owner, scoped credential, kill switch tested
Regulated data with training-use termsDecline until terms changeRevisit on contract amendment
Possible high-risk regulated useHold for counselAssessment resumes with legal classification

Writing the decision down

The output is one page: the use case, the four answers, the risks in scope, the decision, each condition with an owner and a date, the approvers, and a review date. Conditions without owners are wishes. A decision without a review date becomes permanent by accident.

Store it with the inventory entry, so the next person who asks why this was approved can find out in one click. The Record desk covers how decisions like this stay defensible after the fact.

What changes the answer

The model or vendor changes. Re-run questions 2 and 4. A new model version can change failure modes; new terms can change data exposure.

Scope grows. A pilot for ten people approved with manual review does not cover the same tool for two thousand. Volume is a material change.

An incident or near miss. Re-assess immediately and record what the assessment missed.

Next steps

  1. Use the four questions on the next tier 3 request, and time how long the assessment takes.
  2. Keep the twelve-risk table as a scoping step, not a score.
  3. Store the one-page decision with the inventory entry and set a review date.
  4. For technical testing of the system itself, see threat-modelling AI systems and red-teaming an LLM.
  5. Record the outcome with the generative AI risk assessment template.