Best practices for testing AI agents

<p id="docs-hc-snippet" style="display: none;">
Zendesk AI agents are AI-powered chatbots that interact with your customers
on messaging, email (including API and web form), and voice (EAP) channels.
They can have conversations with customers and perform actions in authorized
systems autonomously, automating and resolving issues so that your human
agents can focus on more complex work.
</p>
<div class="article-banners">
<div id="docs-wmp-link">
<a href="https://support.zendesk.com/hc/en-us/articles/5411234991258-plan" rel="noopener" target="_blank">What's my plan?</a>
</div>
<div class="article-banner" id="suite_all">
<table style="margin-left: 0px; margin-right: auto;">
<tbody>
<tr>
<td>
<strong>All Suites</strong>
</td>
<td>Team, Growth, Professional, Enterprise, or Enterprise Plus</td>
</tr>
</tbody>
</table>
</div>
<div class="article-banner" id="support_all">
<table style="margin-left: 0px; margin-right: auto;">
<tbody>
<tr>
<td>
<strong>Support</strong>
</td>
<td>Team, Professional, or Enterprise</td>
</tr>
</tbody>
</table>
</div>
</div>
<p>
Testing your AI agent is a very important step in your onboarding journey to
ensure your <span style="font-weight: 400;">AI agent</span> is ready to graduate
and become a fully functioning member of your customer support team.
</p>
<p>
To ensure you are ready to launch your test should include the following phases:
</p>
<ul>
<li data-list-item-id="e937c0f9b94b75b3c6136ec0b1355c06f">
<a href="#h_01GEHQ7GPVRX76N1RB6D8WTQ48" target="_self">Prepare the Test Environment</a>
</li>
<li data-list-item-id="ebb0ebe835933bcfba335cffad7b446ca">
<a href="#h_01GEKMKKJEK5WGV2AZZ5N5ZYPE" target="_self">Prepare your Testers</a>
</li>
<li data-list-item-id="e861cb7238baaa35f9b9c349cdcbcac60">
<a href="#h_01GEKP9SW4HZCSY15EPFD58ZMM" target="_self">Collecting and Iterating on Feedback</a>
</li>
</ul>
<p id="h_01JKBSFAJXYCC7H4427BZF473D">
<strong>Note</strong> - To do this exercise as a group offline, we have a handout
attached at the bottom of the page.
</p>
<h2 id="h_01GEHQ7GPVRX76N1RB6D8WTQ48">Prepare the Test Environment</h2>
<p>
There are 3 main ways to test your
<span style="font-weight: 400;">AI agent</span>. The first and second options
would have a better outcome, however, the third is available when the others
are not available.
</p>
<ol>
<li data-list-item-id="e41dc111130f772421bdf297c57cf7928">
<a href="#h_01GEKMJKQB7PVSWNTZ6ARYHH0K" target="_self">Connect to CRM and Launch to a Test Environment</a>
</li>
<li data-list-item-id="e662081614b274d35de7ee388b4bc0592">
<a href="#h_01GEKMJT5DWB6ZJ3KB0FE2B3NM" target="_self">Connecting to your CRM with a Test Email Trigger</a>
</li>
<li data-list-item-id="e3f7a997e8a0201cb8e7e7cddcab0a278">
<a href="#h_01GEKMJZS3X3YNE8E90YDJC1HE" target="_self">Using the Test <span style="font-weight: 400;">AI agent</span> button </a>
</li>
</ol>
<h3 id="h_01GEKMJKQB7PVSWNTZ6ARYHH0K">Connect to CRM and Launch to a Test Environment</h3>
<p>
We recommend connecting to your CRM and embedding the widget onto a sandbox,
staging, or testing environment of your website or help center. This will provide
as much of a 'real' experience as possible - where you can assess your
<span style="font-weight: 400;">AI agent</span>'s impact. This means testing
the Actions and Triggers you have set up in their entirety. To ensure your CRM
reporting is not impacted, create a group with a test email to have the tickets
handled separately from the day-to-day.
</p>
<h3 id="h_01GEKMJT5DWB6ZJ3KB0FE2B3NM">Connecting to your CRM with a Test Email Trigger</h3>
<p>
If you don’t have a testing environment you can still test by connecting to your
CRM but will need to be creative as to where you locate the widget or have it
routed off of a rule based on employee email addresses.
</p>
<p>
You can follow the instructions to set this up in Zendesk Support
<a href="https://support.zendesk.com/hc/en-us/articles/8357751811610" target="_blank" rel="noopener noreferrer">here</a>.
</p>
<h3 id="h_01GEKMJZS3X3YNE8E90YDJC1HE">Use the Test AI agent button</h3>
<p>
This is where you can quickly review dialogues and the conversation logs to track
recognition capabilities and quickly assess dialogue structure. We have an article
here on how to use the Test AI agent button
<a href="https://support.zendesk.com/hc/en-us/articles/8357758879130" target="_blank" rel="noopener noreferrer">here</a>.
</p>
<h2 id="h_01GEKMKKJEK5WGV2AZZ5N5ZYPE">Prepare your Testers</h2>
<h3 id="h_01JKBSFAJXPW1QWYNBNRDMHCAH">How do you determine who should be a tester?</h3>
<p>
Ideally, you don't want the same people that built the flows, as they will be
biased based on knowing how the flows are built, and may not be able to see issues
such as typos or formatting if they created them.
</p>
<p>
Then it comes down to finding value for the testers to help - people are more
likely to dedicate time to help test if they understand the value. Therefore
we would recommend 2 types of stakeholders, process-focused and experience-focused
groups. This ensures that you get specific feedback from those respective groups
on two main factors that they are better positioned to offer and have a vested
interest in it. These people can come from wherever in the business, however,
for experience, we think operations and training type team members are best and
for experience, members from social and marketing teams are able to best assess
whether it aligns to brand image, tone of voice and can anticipate what can happen
if the <span style="font-weight: 400;">AI agent</span> makes a mistake.
</p>
<p>
An additional person to include from your support team to help with the validation
would be people who are relatively new to the team. Why?
</p>
<p>
<span style="font-weight: 400;">They have a unique attribute that can make this task especially good for them, which is a fresh set of eyes. They are not influenced by past events and are likely knee-deep in the documentation so can assess where there are discrepancies to ensure everything is up-to-date and correct. In addition to this, they get to learn your company's tone of voice and the way requests will be responded to in the first line of defense - which will accelerate their onboarding process.</span>
</p>
<h3 id="h_01JKBSFAJYAKC4CYB147S08ZR6">
<span style="font-weight: 400;">What do they need to know? </span>
</h3>
<p>
It is important to brief your testers on the intents the
<span style="font-weight: 400;">AI agent</span> understands and what exactly
you are asking them to review. Especially if you ask them to focus on the experience
and tone of voice, you may wish to share with them the persona you built earlier
in the onboarding to ensure they match. Testing can be as robust and detailed
as you like with a team as large or small as you like, but we do have some tried
and tested rules we would recommend you ask your testers to follow.
</p>
<p>
<strong>Golden Rules</strong><br>
❌ Don’t troll the <span style="font-weight: 400;">AI agent</span>. <br>
E.g. “How to take the best selfie with my trainers” <br>
✅ DO simplify your questions and ask one at a time<br>
✅ DO ask the same question in different ways <br>
✅ DO try to get through entire dialog flows and test their different stages and
options (i.e. when the <span style="font-weight: 400;">AI agent</span> fails
to recognize your issue right at the beginning, record this, but then do another
run trying to get past that point and deeper into the dialog flow).<br>
✅ DO take screenshots to accompany your recommendations and feedback.<br>
✅ DO take screenshots of any and all scenarios in which the
<span style="font-weight: 400;">AI agents</span> fail to understand your query
and/or simply deliver a bad user experience.
</p>
<h2 id="h_01GEKP9SW4HZCSY15EPFD58ZMM">Collecting and Iterating on Feedback</h2>
<p>
Collecting feedback is the most important part of the testing process. The more
detailed the feedback the easier the
<span style="font-weight: 400;">AI agent</span> builders' lives will be easier,
first to make improvements but also they can quickly add more expressions based
on how each of your testers would ask about that topic. This is all with the
end goal of getting your <span style="font-weight: 400;">AI agent</span> ready
to be set live.
</p>
<p>
The feedback should be collected with screenshots, on whether the response was
accurate, matched the persona tone, and if it met the expectations to identify
areas for improvement. <br>
You will want to create a shared file with your testers and your builders so
that they can track the progress of the test, see feedback from others to perhaps
upvote and save duplications, and track whether a resolution has been implemented.
</p>
<p>
In addition to this, your builders can review the conversations that happened
in the Conversation Logs to ensure everything happened as intended, such as actions.
</p>
<p>
Once all feedback has been reviewed and you feel happy with your
<span style="font-weight: 400;">AI agent</span>'s performance, they are ready
to be launched and become a full member of your support team.
</p>
<p> </p>

<script id="docs-schema-markup" type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "TechArticle",
"@id": "https://support.zendesk.com/hc/en-us/articles/8357751802138",
"url": "https://support.zendesk.com/hc/en-us/articles/8357751802138-Best-practices-for-testing-advanced-AI-agents",
"headline": "Best practices for testing advanced AI agents",
"description": "You can test your advanced AI agent by preparing a test environment, selecting unbiased testers, and collecting detailed feedback to ensure it performs accurately and aligns with your intended persona before launch.",
"articleSection": "Testing advanced AI agents",
"datePublished": "2024-11-11T22:28:22+00:00",
"dateModified": "2025-09-05T23:48:19+00:00",
"inLanguage": "en-US",
"mainEntityOfPage": {
"@type": "WebPage",
"@id": "https://support.zendesk.com/hc/en-us/articles/8357751802138"
},
"author": {
"@type": "Organization",
"name": "Zendesk",
"@id": "https://www.zendesk.com"
},
"publisher": {
"@type": "Organization",
"name": "Zendesk",
"@id": "https://www.zendesk.com"
},
"isPartOf": {
"@type": "WebSite",
"name": "Zendesk Help",
"@id": "https://support.zendesk.com/hc/"
},
"about": {
"@type": "SoftwareApplication",
"name": "Zendesk",
"@id": "https://www.zendesk.com",
"softwareAddOn": {
"@type": "SoftwareApplication",
"name": "AI agents",
"@id": "https://support.zendesk.com/hc/en-us/sections/8988994355994"
}
}
}
</script>