P1
ProvenanceOne

AI Compliance Review Agent Playbook

Quarterly compliance reviews are labour-intensive: pulling evidence from a dozen systems, cross-referencing controls, documenting gaps. This playbook describes how to build a ProvenanceOne agent that automates the mechanical parts — evidence collection, policy evaluation, gap identification, and structured report generation — and routes all findings to a qualified compliance professional for review and sign-off. Estimated setup time is 2–3 hours.

Warning: This agent assists with compliance evidence gathering and preliminary risk identification. It does not provide legal advice, regulatory interpretation, or final compliance determinations. All findings must be reviewed and signed off by a qualified compliance professional or legal counsel before being relied upon. Do not use agent output as authoritative regulatory guidance.

What this agent does

Compliance review is time-consuming and document-heavy. This agent handles the preliminary, mechanical work:

  1. Collects evidence — a data step queries Snowflake and audit log APIs to retrieve the evidence set for the review period: access logs, configuration snapshots, change records, and prior findings.
  2. Runs policy checks — a skill step (category: data) evaluates each evidence item against pre-defined control criteria you supply. The skill does not have built-in regulatory knowledge — you must provide the control criteria.
  3. Flags gaps and anomalies — an agent step (category: reasoning, trust: high) reviews the policy check results and identifies patterns, anomalies, and potential control gaps.
  4. Generates a structured findings report — a second agent step produces a structured report: findings, evidence references, risk classifications (low, medium, high), and recommended remediation actions.
  5. Routes by risk level — a logic step branches on risk classification:
    • low risk: creates a Jira ticket via an action step for automated tracking.
    • medium and high risk: routes to an approval step for human review before any action.
  6. Stores the evidence package — a storage step writes the complete evidence set, findings report, run ID, agent version, and timestamp to the ProvenanceOne datastore. This package is your auditable record.
  7. Notifies the compliance team — a notify step posts a summary to the designated Slack channel with a link to the full findings.

Best-fit use cases

  • Quarterly and annual controls review for SOC 2, ISO 27001, or internal audit frameworks
  • Automated evidence collection for access reviews: who has access to what, for how long
  • Policy gap identification against a control set you define
  • Continuous monitoring: run on a weekly schedule to catch drift between formal reviews
  • Pre-audit preparation: collect and package evidence before an external auditor engagement

When not to use this agent

  • Final compliance determinations — the agent cannot certify, attest, or sign off on compliance status. A qualified human must make the final determination.
  • Regulatory submissions — never submit agent-generated output to a regulator without human review and sign-off.
  • Risk acceptance sign-offs — risk acceptance requires human judgment and accountability. The agent can surface a risk; it cannot accept it.
  • Client-facing compliance attestations — any attestation shared with a customer or partner must be reviewed and approved by your compliance team.
  • Answering "are we compliant?" — the agent identifies evidence gaps against the criteria you provide. Whether those gaps constitute non-compliance is a legal and professional judgment your qualified team must make.

Required connections and data sources

ConnectionPurposeAuth method
SnowflakeEvidence collection: audit data, access logs, configuration recordsService Account
JiraRemediation ticket creation for low-risk findingsAPI Key
SlackCompliance team notifications and summary deliveryOAuth 2.0
ServiceNow (optional)Alternative or supplementary ticketing for medium/high findingsService Account

Configure connections at Settings → Connections. The Snowflake service account requires read-only access to the audit and logging schemas relevant to your control set. Do not grant write access. See /docs/connections/index.

The following system prompt is a starting point. Adapt the control references to your framework.

You are a compliance evidence analysis assistant. Your role is to review the evidence provided and identify potential gaps against the control criteria supplied to you.

Rules you must follow:
1. Every finding must reference the specific evidence item (query result, log entry, document) that supports it. Do not make assertions without evidence.
2. You cannot make legal determinations. You cannot certify compliance. You cannot advise on regulatory interpretation. If asked for a legal opinion, respond: "This requires review by a qualified compliance professional or legal counsel."
3. Risk classifications (low, medium, high) are preliminary and must be validated by the compliance team before any action is taken.
4. If evidence is ambiguous or incomplete, flag the gap as "evidence insufficient" rather than defaulting to a risk classification.
5. Do not include personally identifiable information in findings summaries or Jira ticket descriptions. Refer to user IDs or anonymised references.
6. Your confidence score reflects the quality and completeness of the evidence reviewed — not a determination of compliance status.

Required skills and tools

StepKindDescription
Policy check evaluatorskillCategory: data. Evaluates evidence items against control criteria JSON you supply. Input schema: { evidence: EvidenceItem[], controls: ControlCriteria[] }. Returns pass/fail/gap per control.
Evidence collection querydataSnowflake query returning audit records, access logs, and configuration snapshots for the review period. Parameterised by review_period_start and review_period_end.
Jira ticket creatoractionCreates a Jira issue with finding summary, risk level, and evidence reference for low risk findings.

The policy check skill must be populated with your control criteria before the agent is useful. Without defined criteria, the agent has nothing to evaluate against. See /docs/skills/index for skill configuration.

[Trigger: schedule (quarterly) | manual | webhook]
       ↓
[Data: collect evidence — Snowflake audit queries]
       ↓
[Skill: policy check — evaluate evidence against control criteria]
       ↓
[Agent: identify gaps, anomalies, patterns — reasoning, trust: high]
       ↓
[Agent: generate structured findings report with risk classifications]
       ↓
[Logic: branch by risk level]
       ↓ low                              ↓ medium / high
[Action: create Jira ticket]    [Approval: compliance officer review]
       ↓                                  ↓ approved / rejected
[Storage: evidence package]    [Action: create Jira / ServiceNow ticket]
       ↓                                  ↓
[Notify: Slack summary to compliance team]

Human approval rules

All medium and high risk findings require an approval step before any downstream action. This is a hard requirement — do not remove approval gates from medium or high risk paths.

Example approval configuration for a high-risk finding:

action: "Review access logging control gap — 47 unlogged API calls identified"
summary: "Policy check identified a potential gap in CC7.2 access logging. 47 API calls in the review period have no corresponding audit log entry. This requires compliance officer review before a remediation ticket is filed."
risk: high
slaMinutes: 480
assignees:
  - [email protected]
  - [email protected]
rationale: "Control gap identified in access logging. 47 unlogged API calls in last 30 days. Evidence insufficient to determine whether gap is a system fault or a policy violation."
confidence: 0.78
evidence:
  - label: "Framework"
    value: "SOC 2 CC7.2"
    tone: amber
  - label: "Unlogged events"
    value: "47"
    tone: red
  - label: "Risk level"
    value: "High"
    tone: red
  - label: "Review required"
    value: "Legal sign-off needed before external reporting"
    tone: amber

SLA guidance:

Risk levelRecommended SLA
lowAutomated — no approval required
medium1,440 minutes (24 hours)
high480 minutes (8 hours, same business day)

Never automate:

  • Final compliance determinations
  • Regulatory submissions
  • Risk acceptance sign-offs
  • Client-facing compliance attestations
  • Any conclusion presented to regulators without human review and sign-off

Security and permission model

Snowflake access — the service account used in the data step must have read-only access, scoped to the schemas required for the review. Rotate credentials on a schedule. The connection secret is stored in the secrets vault; access emits secret.accessed to the audit log.

Evidence datastore access — only admin and designated editor roles on the compliance team should have access to the evidence datastore. Do not grant viewer access to the evidence package datastore to the broader organisation.

DLP on agent outputs — configure a Gateway DLP policy to prevent PII from appearing in agent-generated findings summaries, Jira ticket bodies, or Slack notifications. Reference user IDs, not names or email addresses, in findings.

Approval authority — assign approvals to named individuals in the approvers platform group. Do not use a shared team inbox — approval actions are individually attributed in the audit log.

Roles:

RoleCan do
adminCreate and modify agent and workflow, publish to production, access evidence datastore
editorModify workflow, update control criteria in policy check skill
viewerView run history and findings report — cannot modify configuration or action approvals
approvers groupAction approval requests for medium and high risk findings

Audit events emitted: run.started, run.completed, run.failed, approval.granted, approval.rejected, approval.sla_breach, agent.updated, connection.accessed, secret.accessed, workflow.published.

The complete audit log — including every step, tool call, approval action, actor ID, and timestamp — is stored in ProvenanceOne's audit service, signed with HMAC-SHA256. See /docs/audit/index.

Evaluation checklist

  • Every finding references the specific evidence item (query result, log entry) that supports it — no ungrounded assertions
  • System prompt explicitly states the agent cannot make legal determinations or certify compliance
  • Every medium and high finding routes to an approval step before any action — test the logic step branching
  • Evidence package stored in the datastore includes: run ID, agent version, timestamp, and the full evidence set
  • A qualified compliance officer has reviewed and confirmed the accuracy of findings from at least one test run against known controls
  • DLP policy prevents PII appearing in agent outputs, Jira tickets, and Slack notifications — test with a known PII-containing evidence set
  • Audit log confirms the full chain: data retrieval → agent reasoning → approval → action for at least one end-to-end test run
  • The policy check skill is populated with your specific control criteria — do not run against an empty control set
  • False positive rate is tracked for the first four weeks of operation

Rollout plan

Phase 1 — Shadow mode: Run the agent in parallel with your existing manual review process. Do not use agent output to drive any action. Compare the agent's evidence coverage and gap identification to the manual review. Identify mismatches and refine the policy check skill criteria.

Phase 2 — Evidence collection only: Use the agent to collect and package evidence. Your compliance team performs all analysis manually. This phase validates that the data step retrieves the correct evidence sets and that the storage step produces a complete, auditable package.

Phase 3 — Preliminary risk flagging: Use the agent for risk classification of low-complexity findings. Your team reviews all flagged items before any action. Measure the false positive rate and confidence score calibration.

Phase 4 — Differentiated flow: Use the agent end-to-end for low risk findings with quarterly spot-check reviews. medium and high findings always require human approval before any action. This remains the steady-state operating model — do not remove the approval gates from medium and high risk paths.

Common failure modes

Agent flags false positives The policy check skill produces gap results for controls that are actually satisfied, overwhelming the compliance team with noise. Mitigation: tune the policy check skill with a baseline of known-good evidence from a prior review. Track false positive rate weekly during the first month. Tighten control criteria definitions.

Evidence gaps from unconnected systems The agent cannot access a system that holds relevant evidence (e.g. a legacy access management tool not connected to Snowflake). The report implies coverage it does not have. Mitigation: explicitly document which systems are covered and which are not in the findings report. Never present the report as representing full coverage. Add a "systems not covered" section to the report template.

Compliance officer approves without reading the evidence The approval is actioned without genuine review, defeating the purpose of the gate. Mitigation: surface the key evidence items in the approval payload — use tone: red and tone: amber to draw attention to the most significant findings. Consider requiring approvers to add a comment confirming their review action before approval is processed.

Control criteria out of date The policy check skill evaluates against last year's control set after a framework update. Mitigation: version-control the control criteria JSON. Add a last_reviewed date to the skill configuration. Schedule a quarterly review of the control criteria alongside your policy review cycle.

ROI assumptions

The table below uses illustrative assumptions. Replace with your organisation's actual values. Do not estimate "audit finding escape rate reduction" without baseline data — track it post-implementation.

InputIllustrative value
Reviews per quarter4 (quarterly controls review)
Hours per review — current (manual)40 hours
Hours per review — with agent15 hours (agent handles evidence collection; human performs analysis)
Hours saved per review25 hours
Loaded hourly cost (GRC analyst)$90
Annual time savings4 reviews × 25 hours × $90 = $9,000
Audit finding escape rate reductionTrack post-implementation — do not estimate without baseline data

Use the interactive calculator to adjust these inputs: /tools/ai-agent-roi-calculator?use_case=compliance-review

FAQ

Can this agent certify our SOC 2 compliance?▾

No. The agent assists with evidence gathering and preliminary gap identification. SOC 2 certification requires an engagement with a qualified, accredited auditor. Agent output can support audit preparation — organising evidence, flagging potential gaps before the auditor arrives — but it does not replace the audit process or produce a certifiable result.

What frameworks does the agent support?▾

The agent evaluates evidence against whatever control criteria you define in your policy check skill. It does not have built-in knowledge of SOC 2, ISO 27001, GDPR, HIPAA, or any other framework. You must supply the control criteria as structured input to the skill. This means the agent is framework-agnostic, but it requires upfront work to define and maintain accurate criteria.

How do I prevent the agent from making a false compliance claim?▾

Two mechanisms: system prompt and approval gates. The system prompt must explicitly state that the agent cannot make final determinations and cannot certify compliance. Every medium and high risk finding must pass through an approval step before any external communication or action. Do not remove these gates.

Is the agent's audit trail admissible as compliance evidence?▾

ProvenanceOne's audit log records every run, step, tool call, and approval with timestamps and actor IDs, signed with HMAC-SHA256. Whether this satisfies a specific framework's evidence requirements — for example, as a control activity log — depends on the framework, the specific control, and your auditor's judgment. Confirm with your compliance team and external auditor before relying on the audit log as primary evidence for a specific control.

Can I run this weekly instead of quarterly?▾

Yes. Change the trigger type from schedule (quarterly) to schedule (weekly) and adjust the evidence window in the data step to cover the last 7 days. Running weekly allows you to catch drift earlier and reduces the evidence backlog before a formal quarterly review. The storage step accumulates evidence continuously — adjust your evidence package retention policy accordingly.