How to Test an AI Tool Before Using It for Client Work
An AI demonstration usually shows what a product can do. Before we use it for client work, we need to understand what it should do, where it struggles and how we will recognise a problem.
That does not mean every accounting firm needs a testing laboratory. It means approving a defined use, supported by evidence, rather than approving an entire product because one example looked impressive.
On 13 November 2026, Anthropic published guidance on stronger safety evaluations, including the importance of documenting the configuration being tested. It is supplier-authored research, not proof that any particular accounting workflow is safe.
We can nevertheless use that principle constructively. Here is our recommended approach to a proportionate accounting-firm pilot.
- DefineTask, data & limits
- TestOutputs & permissions
- ReviewPeople & evidence
- ReassessChanges & incidents
1Approve a specific job, not a product name
“Approve Copilot” or “approve our bookkeeping AI” is too broad.
Write down the task, the intended user, the information involved and the permitted output. For example:
Hypothetical scope: prepare draft explanations of monthly overhead movements from an approved management-accounts pack. The tool must not send reports, change ledger entries or provide tax advice.
That statement gives us something we can test.
Also record what success would mean. Faster drafting might be worthwhile, but only if the final explanations are accurate and review does not consume the time saved.
If your firm has not yet mapped its AI use, our AI governance self-assessment can help identify the starting gaps. It is a readiness check, not certification.
2Establish the data boundary before testing
Begin with synthetic information where possible. If realistic examples require client information, resolve the confidentiality, contractual and data-protection questions first.
ICAEW’s guidance on AI and client confidentiality reminds firms that professional confidentiality obligations continue to apply when using AI. A pilot is not an exemption.
Our recommended pre-test questions are:
- Which provider and subprocessors receive the information?
- Where is it processed and retained?
- Can inputs or outputs be used for model improvement?
- Which users and connected applications can access it?
- How will test information be deleted?
Record unanswered questions. If an answer is necessary to establish whether the proposed data use is acceptable, do not resolve the uncertainty by uploading the data anyway.
This guide is an operational testing method, not a substitute for assessing applicable legal or professional duties.
3Build a small but demanding test pack
Use examples that reflect the intended work, not just clean examples supplied for demonstrations.
For a draft commentary tool, we would include an ordinary month, a missing comparative, an unusual credit, inconsistent department labels and a movement with more than one plausible explanation.
Prepare the expected answer or acceptable response before running the test. Sometimes the correct response is a question: “We need the supporting schedule before explaining this movement.”
Have an experienced accountant check the answer key. Otherwise, an incorrect benchmark can make a good response look wrong—or reward a convincing mistake.
The NIST AI Risk Management Framework Playbook recommends evaluating performance in conditions resembling deployment and documenting the measures. This is voluntary guidance, not a UK statutory testing requirement.
A modest pilot helps identify failure modes. It does not establish a statistically reliable accuracy rate for every client.
4Decide what would stop the pilot
Agree acceptance criteria before seeing the results.
We would separate three types of outcome:
- Unacceptable: unauthorised access, an unapproved external action or a material unsupported conclusion presented as established fact.
- Needs correction: an omission or categorisation error that the review process identifies and resolves.
- Useful behaviour: a correct result, a clear limitation or a request for missing information.
Do not allow an attractive average score to hide a serious failure. Ninety-nine satisfactory summaries would not justify one unauthorised client disclosure.
The thresholds should reflect the task. A tool suggesting internal email wording does not need the same controls as a workflow preparing entries for posting.
Write down who can pause the pilot and what evidence is needed before it restarts.
5Test permissions and actions separately from answers
A correct answer does not prove that access controls work.
Use authorised test accounts representing different roles. Check whether each can retrieve only the records intended for that role. Test separation between clients using synthetic records with clearly distinguishable names.
If the tool can take actions, examine each action independently. Drafting an invoice, sending it and recording a payment are different permissions.
Try an unapproved action within the agreed test environment. Confirm that the system blocks it or requires the expected approval, and that the event appears in the relevant log.
Keep this testing within your permissions and the supplier’s terms. Do not probe other customers’ data or attempt to defeat security controls on a live service.
6Test the reviewer, not just the AI
“Human review required” is only useful if the reviewer can do the job.
Give the reviewer the underlying records, output and relevant context. Ask them to identify deliberately seeded errors in a synthetic exercise.
Can they trace figures back to their source? Can they distinguish an evidenced explanation from a plausible suggestion? Do they know how to reject the output?
Measure review and correction time as well as generation time. A draft produced in seconds may still create more work than it removes.
We would also avoid relying solely on a second AI system to approve the first. Automated checking can assist, but the firm should understand what that checker measures and what it could miss.
7Keep a decision record that someone else can understand
Retain the approved scope, test inputs, expected results, actual outputs, defects and reviewer conclusions. Protect this evidence according to its sensitivity.
Record the product, account type, available model identifier, settings, connectors and test date. If a supplier does not expose a model version, record that limitation instead of guessing.
NIST’s AI Risk Management Framework, first released in 2023, provides a voluntary structure for managing AI risks. Our practical recommendation is to turn that discipline into a short approval record, not a folder of unexplained screenshots.
Approval should say: approved for this task, with these users, these controls and these exclusions.
A draft variance-commentary assistant
The following is hypothetical, not a product test.
A practice tests 24 synthetic monthly-reporting cases. Twenty produce acceptable drafts. Three need minor corrections. One confidently attributes higher costs to recruitment, although the underlying pack contains no evidence for that explanation.
The reviewer catches the unsupported statement, but only by checking the source schedules.
We would not describe this as proof that the tool is “83% accurate”. The sample is small, deliberately constructed and covers only one task.
Instead, the practice could revise its instructions to separate evidenced explanations from questions for the client, retest the failed case and add further unsupported-causation examples.
Any approval would remain limited to draft preparation with source-based review—not autonomous client reporting.
8Set the conditions for another test
Approval should not silently extend to new capabilities.
We would trigger reassessment when the supplier changes a model or material feature, the firm connects another data source, permissions expand, or a significant error or complaint occurs.
For a low-risk drafting tool, that may mean rerunning a short test pack. A workflow gaining permission to post transactions needs a fuller review.
Our framework pathway provides the wider governance context. The pilot should support that process, not sit apart from it.
Before approving, confirm:
- The task and exclusions are clear.
- Data use and permissions are acceptable.
- Representative and difficult cases have been tested.
- Review works in practice.
- An accountable person has recorded the decision and retest triggers.
Start with one workflow. Record its owner in the free AIGAS AI Register and keep the evidence alongside your approval records.
The aim is not to prove that AI never makes mistakes. It is to establish where it is useful, how problems will be detected and who remains responsible.
Sources & further reading
- Anthropic: Building strong safety evaluations — supplier-authored evaluation guidance, not validation of an accounting workflow.
- ICAEW: AI and client data — complying with the principle of confidentiality.
- NIST AI RMF Playbook and AI RMF Core, Measure 2.1–2.3 — documented tests and evaluation in deployment-like conditions.
- NIST AI Risk Management Framework — voluntary guidance, first released in 2023.
