How Should Enterprises Think About Finance AI Safety and Governance?
This report is the introduction to a series of research notes dealing with issues of Finance AI safety. How enterprises can leverage Gen AI and agentic capabilities while carefully managing for precision, predictability, consistency, and compliance is explored. The framework presented helps finance leaders understand what functions and processes can be self-driving, which require oversight, and which should not be addressed with AI. This report walks through the questions finance leaders need to ask, what their responses indicate, and offers guidance on maximizing AI’s utility while minimizing risk.
Ralph Nader’s seminal 1965 book, Unsafe at Any Speed: The Designed-In Dangers of the American Automobile, reshaped how society thinks about safety, responsibility and regulation. In the 1950-60s, the auto industry prioritized style, speed, and marketing over safety. Cars were getting more powerful and esthetically appealing, but there were few safety standards, and rising highway deaths were often blamed on driver error rather than vehicle design.
Ralph Nader challenged this narrative. He argued that while driver error is a factor, automakers knowingly ignored safety features that could save lives. The book famously focused on the Chevrolet Corvair, a rear-engine car made by General Motors, putting a spotlight on the lack of crash-testing and safety standards in the automotive industry. The book had enormous real-world consequences, sparking public debate, congressional hearings, and creation of new regulatory bodies and standards, increasing consumer awareness about safety as a design attribute. Adoption of seatbelts, collapsible steering columns, and other safety features were catalyzed by Nader’s book.
Flashing forward to today, there is a tremendous push for adopting AI to finance automation. New start-ups claiming to be “AI-first” are challenging the status quo of established processes, benchmarks, and category boundaries. The upside is clear: finance teams are already resource-constrained, and Gen AI capabilities can help automate roughly 30-40% of finance workflows, freeing up valuable time and resources. MGI Research estimates that approximately 95% of finance professionals are either using or want to use Gen AI capabilities in their financial automation processes.
Despite the upside of AI in finance, there is also notable risk. Gen AI-based solutions, including agentic AI, are probabilistic and routinely make mistakes. The finance function values precision, predictability, consistency, and compliance over speed. Given the same inputs and identical context, finance teams expect an exact and identical answer for every iteration at any time. Gen AI, together with dirty data, can magnify small errors multi-fold. Without a governance framework and a set of practiced standards, tiny mistakes become huge problems. The risks are real and material. Loss of confidence in company financials, restatements, failed audits, personal and corporate legal responsibility with long-term reputational damage are just a few. Meanwhile, leading LLMs today all include the simple disclaimer, “AI can make mistakes,” on their web portals. In the world of “AI-first” finance automation this message is often lost or muffled.
Can AI-Generated Output be Controlled, Repeatable, and Auditable?
Unlike traditional deterministic systems, AI models introduce probabilistic behavior, creating challenges around explainability, data lineage, and verification. This has elevated concerns around hallucinations, silent errors, and context loss – unacceptable risks in finance. Consequently, enterprises are converging on an approach that treats AI as an extension of model risk management frameworks. Similar to credit or market risk models, AI systems are increasingly subject to validation, monitoring, and periodic re-certification, with explicit governance over where and how they can be deployed.
Governance capabilities are becoming a key gating factor for adoption of AI in the Office of the CFO. Leading organizations are establishing formal AI risk frameworks defining use-case eligibility, enforcing human-in-the-loop controls for material decisions, and requiring comprehensive audit logs of system behavior. In practice, this results in a tiered deployment model: lower-risk use cases such as classification, anomaly detection, and workflow assistance are prioritized, while higher-risk applications like autonomous journal entries or regulatory reporting remain tightly constrained. Parallel runs, reconciliation against ground truth, and sandboxed environments are becoming standard operating procedures, reflecting a broader cultural reality that trust in AI must be earned over time rather than assumed.
A critical emerging dimension is the rise of agentic AI: systems that can generate insights and take actions across financial systems. This introduces a qualitatively different risk vulnerability, including unauthorized transactions, breakdowns in approval workflows, and loss of control. In response, enterprises are implementing increasingly granular permissions, identity management for non-human actors, and fail-safe mechanisms such as kill switches and rollback capabilities. Notably, this is exposing gaps in existing security architectures, which were not designed to accommodate autonomous software agents operating with delegated authority.
AI Autonomy (AIA) Framework for Governance
Underlying all these considerations is a more prosaic and decisive constraint: data governance. The effectiveness and safety of AI systems are directly tied to the quality, integrity, and control of underlying financial data. Weaknesses in data lineage, access controls, or consistency can propagate rapidly through AI-driven workflows, undermining both accuracy and auditability. Many organizations are finding data infrastructure – not AI model sophistication – is the primary bottleneck to scaled adoption, especially in organizations experimenting with usage-based pricing models.
“Safe AI” adoption in finance is coalescing around a four-point operating model: clear guidelines on what can and cannot be done with AI in finance, governed autonomy, transparency, and human accountability. Organizations that succeed at articulating a simple set of rules for AI in finance will dramatically lower the risk of the spread of “shadow AI” when well-intentioned grassroot AI adoption can lead to critical data breaches.
MGI Research created the AI Autonomy (AIA) framework addressing the right level of human involvement in Finance AI. This framework consists of five distinct stages outlining how self-driven a specific finance process and capability can be. Determining the right level of autonomy requires finance leaders to ask several key questions of each process seeking automation.
The Five Levels of AI Autonomy
Level 1 (Not a Fit for AI): AI involvement adds risk without meaningful benefit. Three distinct triggers identify these functions. Any one is sufficient to categorize the function as human only.
Level 2 (AI-assisted): AI informs, drafts, or surfaces insight. The human does the work. Appropriate when there is no objective grading standard.
Level 3 (Human-in-the-Loop): AI recommends, a human approves before execution. Required when financial exposure is material or regulatory obligations apply.
Level 4 (Human-on-the-Loop): AI acts, a human can override before consequences become irreversible. Override window must be shorter than reversal window.
Level 5 (Fully Autonomous): AI acts without human review. Errors can be undone cheaply, accuracy is validated, failure spread is controlled.
Showstoppers: Three Triggers Permanently Rule Out Autonomous AI
Most governance frameworks treat “not a fit for AI” as a single undifferentiated category. This framework distinguishes three independent “showstopper” triggers, each with different legal, governance, and communication implications. An application of AI implicating any one trigger belongs at Level 1 (Human Only) regardless of AI capability now or in the future.
Fiduciary and Legal Accountability: A named individual is legally accountable to a third party for the conclusion. Examples are a CFO attesting under SOX or a General Counsel approving contract risk. Rubber-stamped AI output may satisfy the process on paper but omits the degree of human control exercised without eliminating personal liability.
High-Stakes Relational: The customer’s experience of being dealt with by a human is part of the value exchange. Strategic renewals, billing disputes with key accounts, and exception deal negotiations are clear examples. Automating these exchanges signals to the customer that they are a transaction, not a relationship.
Market Integrity: AI outputs flowing into publicly reported financial results affect investors and market participants who are not party to the original decision. No single company’s accountability framework can contain this harm. Revenue recognition positions and earnings-affecting estimates belong in this no-autonomy trigger category.
Picking the Correct AIA Level: Nine Sanity Questions
To help organizations decide how much AI autonomy to give a particular process (what AIA level to pick) organizations should pressure-test each process by asking nine critical questions. These questions are designed to be a guide rather than a hard-boiled rule cookbook. Each question is evaluated for every AI sub-function. Questions 1, 2, and 3 are barometers where a negative answer to any of these can determine the autonomy level before remaining questions are assessed. If there is hesitation, the most restrictive AI autonomy level governs.
Question 1: Reversibility – Can it be undone cheaply if it’s wrong?
Not whether reversal is theoretically possible, but whether it is low friction, low cost, has no customer impact, nor any audit consequence. In a typical business process pipeline, reversibility decreases at each downstream step and errors compound. Autonomy level misclassification early in the process propagates forward, making the cost of correction higher than any individual step suggests.
Executive Decision Rule: If undoing an error requires a manual remediation process, customer communication, financial adjustment, or an auditor conversation, treat the action as irreversible when determining the autonomy level.
Example: Tagging a contract as under-review is reversible. Sending an invoice is not reversible without an explanation to a customer as an exception. Applying the correction at scale cannot be easily reversed.
Question 2: Financial Exposure – What does one mistake cost? What does a systematic mistake cost?
Evaluate both per-transaction magnitude and aggregate systematic exposure. Per-transaction exposure sets the autonomy floor. Aggregate exposure determines whether this floor is sufficient. Sample starting points: under $1K per transaction allows full automation; $1K–$25K requires a human override window; above $25K requires human approval before execution.
Executive Decision Rule: If a threshold is not written down, it does not exist. In a post-incident review, the absence of a written policy is its own liability. Calculate both per-transaction and aggregate systematic exposure.
Example: Auto-applying an approved volume discount at sub-$1K exposure can be fully autonomous if other dimensions agree. A revenue recognition adjustment on a $5M multi-element arrangement requires human approval before execution. Billing reconciliation errors individually small but systemically large means calculating both.
Question 3: Regulatory/Personal Accountability Exposure – Is someone personally liable for this?
The critical distinction is between two types of controls. Automated application controls like deterministic, rule-based logic that can be audited/verified are recognized and, in many cases, preferred by accounting regulators. AI can legitimately power these. Management review controls are different: they require a named individual to own the conclusion and attest to it. For public companies, revenue recognition positions, variable consideration estimates, and period-close attestations are management review controls subject to CEO and CFO certification under SOX 302 and 906. AI can prepare the analysis. It cannot discharge the attestation.
Executive Decision Rule: Automated application controls can be AI-driven with appropriate logging. Management review controls require human approval before execution as the minimum. No exceptions. Autonomy between Levels 1 and 3.
Example: AI calculating a standalone selling price (SSP) using a documented deterministic methodology is a legitimate automated control if auditable. A CFO attesting to the revenue recognition position in a 10-Q requires management review control. The attestation must be human regardless of what AI prepares.
Question 4: Precision Tolerance – How accurate does it need to be?
Precision requirements are frequently asymmetric: false positives and false negatives carry different costs and must be evaluated separately. Some functions have near-zero tolerance in both directions. These require rule-based automation with AI in an exception-detection role rather than a primary decision role.
Executi...
