Frontier AI Security Checklist

A frontier AI security checklist helps practitioners confirm whether each AI system is visible, governed, monitored, and containable. Use it during architecture review, vendor review, predeployment approval, red-team planning, and quarterly control validation.

Inventory

Unknown AI use creates unmanaged risk before a single prompt runs. A strong inventory should reveal the business process, owner, trust boundary, data path, and operational dependency behind each AI system.

• Models: Document every approved, embedded, and externally accessed model, including provider, version, owner, business purpose, hosting model, and risk tier.
• Agents: Identify every agentic workflow, including what goal it serves, which tools it can call, which identity it uses, and which actions it can execute.
• Copilots: Record enterprise copilots across productivity suites, developer platforms, CRM, ITSM, security tools, HR systems, finance systems, and customer support.
• APIs: Inventory model-access APIs, API keys, model gateways, orchestration services, and applications that send data to external AI services.
• Vector stores: Track vector databases, retrieval indexes, embedding models, source corpora, data owners, access controls, and retention rules.
• Plugins and connectors: List SaaS connectors, browser extensions, plugins, tool integrations, OAuth grants, and service accounts connected to AI workflows.
• SaaS AI features: Review AI features introduced through vendor updates, especially summarization, semantic search, autonomous routing, assistant interfaces, and agent builders.

Data

Frontier AI changes data protection because information can move through prompts, embeddings, retrieval layers, memory, logs, and generated responses in the same workflow. Security teams need to protect data as it enters context, persists in supporting systems, and reappears in outputs.

• Prompts: Define which data classes users may include in prompts, uploads, pasted content, screenshots, code snippets, and task instructions.
• Uploads: Scan files for regulated data, credentials, customer records, proprietary code, confidential documents, hidden instructions, and malicious content.
• Embeddings: Protect embeddings with encryption, access control, retention limits, tenant isolation, and ownership records.
• Logs: Configure AI logs to support investigation without retaining unnecessary sensitive data.
• Memory: Govern persistent memory with visibility, deletion paths, administrative policy, and limits on sensitive data retention.
• Retrieval stores: Enforce entitlement-aware retrieval at query time, not only during indexing.
• Outputs: Inspect generated text, code, summaries, recommendations, and tool instructions for leakage, unsafe guidance, and unsupported claims.

Identity

AI security fails when conversational interfaces obscure who holds authority. Every AI-enabled workflow needs clear separation between user access, machine access, agent access, and inherited connector permissions.

• Human users: Apply role-based access, step-up authentication, approved-use policies, and logging for users of high-risk AI systems.
• Service accounts: Replace shared or persistent credentials with scoped, time-bound identities wherever possible.
• Agent credentials: Give agents the narrowest authority needed for the task, with separate identities from the human user.
• Connector permissions: Review OAuth scopes, SaaS grants, cloud roles, repository access, ticketing privileges, and email or file access.
• Revocation paths: Maintain a tested process to revoke AI-connected credentials, keys, tokens, and grants quickly.
• Privilege drift: Monitor agent and connector permissions for expansion after product updates, workflow changes, or role changes.

Related Article: Implementing Frontier AI Security: A Roadmap

Actions

The moment AI can act, security must govern consequence rather than interface. A harmless-looking request can become a production change, customer communication, code commit, or privileged workflow step if tool access lacks constraint.

• Tool calls: Define which tools each model or agent may call, what inputs each tool accepts, and what data each tool can return.
• Approval gates: Require explicit approval for production changes, code commits, customer communications, financial activity, privileged identity changes, and security control modifications.
• Rollback: Test rollback for every high-impact AI-driven action before deployment.
• Rate limits: Apply rate limits to tool calls, retrieval queries, API requests, external messages, code submissions, and automated workflow execution.
• Dry-run modes: Use dry-run mode for cloud changes, policy updates, code modifications, ticket closures, and other state-changing actions.
• High-impact workflows: Route regulated, customer-facing, financial, production, and critical-infrastructure workflows through stronger review.

Monitoring

AI activity needs observability that captures intent, context, execution, and downstream effect. Without that evidence, teams can’t distinguish safe assistance from prompt abuse, data exposure, unauthorized retrieval, or agentic misuse.

• Prompts: Monitor prompts for sensitive data, jailbreak attempts, prompt injection, policy bypass attempts, and unusual task patterns.
• Outputs: Inspect completions for data exposure, unsafe code, hallucinated evidence, policy violations, and harmful instructions.
• Refusals: Track refusal rates and repeated attempts to bypass refusals through prompt variation.
• Retrieval events: Log retrieved sources, query patterns, index access, sensitive repository use, stale-source use, and permission failures.
• Tool calls: Correlate tool-call telemetry with cloud, SaaS, endpoint, identity, API, source-code, and ticketing events.
• Model changes: Track model versions, system prompt changes, routing changes, tool-schema updates, connector changes, and safety-layer updates.
• SOC integration: Feed AI telemetry into SIEM, SOAR, XDR, CNAPP, identity monitoring, SaaS audit monitoring, and data security workflows.

Evaluation

Frontier AI testing should prove how the system behaves under pressure. Evaluations need to exercise adversarial inputs, ambiguous instructions, permission boundaries, tool misuse, and model-update regressions.

• Prompt injection: Test direct and indirect injection through prompts, documents, web pages, tickets, emails, code comments, images, and tool results.
• Data leakage: Test whether the system exposes secrets, regulated data, customer records, proprietary code, hidden system prompts, or unauthorized documents.
• Excessive agency: Test whether agents can take actions beyond their approved authority, especially through broad credentials or ambiguous tool permissions.
• Jailbreaks: Maintain a regression suite of known jailbreaks, refusal bypasses, and policy evasion attempts.
• Cyber misuse: Evaluate whether the system assists phishing, malware development, exploit generation, credential theft, evasion, or unauthorized persistence.
• Retrieval poisoning: Test whether untrusted, stale, or manipulated documents can alter model behavior or produce unsafe recommendations.
• Regression: Rerun evaluations after model updates, prompt changes, retrieval changes, tool-schema changes, and connector-permission changes.

Governance

Frontier AI requires durable accountability because models, providers, use cases, and controls will keep changing. Governance should define who owns the risk, who can approve exceptions, what evidence must exist, and how unresolved exposure reaches leadership.

• Ownership: Assign every AI system a business owner, technical owner, security owner, data owner, and risk tier.
• Policy: Define acceptable use, approved providers, prohibited data, retention requirements, logging rules, tool permissions, and agent authority.
• Exceptions: Require documented justification, compensating controls, monitoring, expiration dates, and approval authority for exceptions.
• Vendor risk: Review provider training use, retention, isolation, subprocessors, model updates, logging, breach reporting, audit rights, and termination support.
• Auditability: Preserve model version, system prompt version, retrieved sources, tool calls, approvals, outputs, and downstream actions.
• Board reporting: Report AI asset coverage, high-risk systems, unresolved exposure, sensitive data events, prompt injection attempts, blocked tool calls, red-team findings, vendor concentration, and time to revoke compromised agent credentials.

   

Frontier AI Security Checklist FAQs