Evaluating the performance of AI agents using Zendesk QA

<body>
<p id="docs-hc-snippet" style="display: none;">Zendesk QA can help you evaluate how well yourAI agentsperform in conversations with your customers. You can use this information to update your AI agents and workflows based on the results.</p>
<div class="article-banners">
<div id="docs-wmp-link">
<a href="https://support.zendesk.com/hc/en-us/articles/5411234991258-plan" rel="noopener" target="_blank">What's my plan?</a>
</div>
<div class="article-banner" id="addon_zqa">
<table>
<tbody>
<tr>
<td>
<strong>Add-on</strong>
</td>
<td>Quality Assurance (QA) or Workforce Engagement Management (WEM)</td>
</tr>
</tbody>
</table>
</div>
</div>
<div id="docs-ai-summary">
<p id="docs-ai-summary-toggle">Verified AI summary <span id="up-triangle">◀</span><span id="down-triangle">▼</span></p>
<p id="docs-ai-summary-content">Use the QA feature to assess AI agent performance in customer interactions. Configure which bots to evaluate, and use scorecards to review conversations manually or automatically. Analyze results with the Reviews dashboard and monitor key metrics like escalations with the BotQA dashboard. This helps you refine AI workflows and improve customer support quality.</p>
</div>
<div class="body">
<p class="p" dir="ltr">Zendesk QA can help you evaluate how well your <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/6970583409690" target="_blank">AI agents</a> perform in conversations with your customers. You can use
this information to update your AI agents and workflows based on the results.</p>
<p class="p" dir="ltr">This article contains the following topics:</p>
<ul class="ul" id="topic-1__ul_w2h_sv1_cbc">
<li class="li"><a class="xref" href="#topic_wrh_gzs_rdc">Configuring which AI agents are
evaluated</a></li>
<li class="li" dir="ltr"><a class="xref" href="#topic_kyy_xvm_5hc">Evaluating AI agent

conversations</a></li>
</ul>
<p class="p" dir="ltr">Related articles:</p>
<ul class="ul" id="topic-1__ul_x2h_sv1_cbc">
<li class="li"><a class="xref" href="https://support.zendesk.com/hc/en-us/articles/8071540887450" target="_blank">Manually identifying AI agents in Zendesk QA</a></li>
<li class="li" dir="ltr"><a class="xref" href="https://support.zendesk.com/hc/en-us/articles/7418648572826" target="_blank">Using the BotQA dashboard to understand bot escalations and
performance</a></li>
</ul>
</div>
<div aria-labelledby="ariaid-title2" class="topic nested1" id="topic_wrh_gzs_rdc">
<h2 class="title topictitle2" id="ariaid-title2">Configuring which AI agents are evaluated</h2>
<div class="body">
<p class="p" dir="ltr">Zendesk QA automatically detects <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/6970583409690" target="_blank">Zendesk AI agents</a> on messaging channels.</p>
<p class="p" dir="ltr">You can also manually report other users as AI agents so they can be reviewed
using the correct resources. See <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/8071540887450" target="_blank">Manually identifying AI agents in Zendesk QA</a></p>
<p class="p">Bots are included in reviews unless you exclude them. You can configure the review settings for each
bot on the Bots page.</p>
<p class="p" dir="ltr"><strong class="ph b">To configure whether or not a bot is reviewable</strong></p>
<ol class="ol" id="topic_wrh_gzs_rdc__ol_w3q_l51_tcc">
<li class="li" dir="ltr"><span class="ph">In <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/4408838272410" target="_blank">Quality assurance</a>, click your profile icon in the top-right
corner, then select <strong class="ph b">Users, bots, and workspaces</strong>.</span></li>
<li class="li" dir="ltr">Select <strong class="ph b">Bots</strong>.</li>
<li class="li">The list of bots appears, including the following columns:<ul class="ul" id="topic_wrh_gzs_rdc__ul_c2c_ktm_5hc">
<li class="li" dir="ltr">
<strong class="ph b">Bot name: </strong>The name of the bot.</li>
<li class="li" dir="ltr">
<strong class="ph b">Last chat: </strong>When the last conversation with the bot took place.</li>
<li class="li" dir="ltr">
<strong class="ph b">Reviewable:</strong> Whether the bot is included in reviews.<p class="p"><img class="image" id="topic_wrh_gzs_rdc__image_lrl_25m_5hc" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/qa_bots.png" width="850"></p>
</li>
</ul>
Find the bot in the list, then select a value in the <strong class="ph b">Reviewable</strong> column:<ul class="ul" id="topic_wrh_gzs_rdc__ul_vpm_rzs_rdc">
<li class="li">
<strong class="ph b">Yes</strong>: The bot is configured to be reviewed. If <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/7043747123354" target="_blank">autoscoring</a> is turned on, bots are reviewed automatically.
However, you can also <a class="xref" href="#topic_hh4_fnk_thc">review your bots
manually</a>.</li>
<li class="li">
<strong class="ph b">No</strong>: The bot is excluded from reviews, which means it will not be included in
autoscoring or assignments. Non-reviewable bots are also not displayed in <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/7043669455130" target="_blank">filters</a>. Additionally, new data will not appear in <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/7043701144858" target="_blank">dashboards</a>. This option is useful if you lack sufficient
context to evaluate the bot.</li>
</ul>
</li>
</ol>
</div>
</div>
<div aria-labelledby="ariaid-title3" class="topic nested1" id="topic_kyy_xvm_5hc">
<h2 class="title topictitle2" id="ariaid-title3">Evaluating AI agent conversations</h2>
<div class="body">
<p class="p" dir="ltr">You can use Zendesk QA to evaluate the performance of your bots across various
categories just like you can for human agents. </p>
<p class="p" dir="ltr">To do so, you must <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/7043760215194" target="_blank">set up a scorecard</a> for the categories you want to evaluate the bot
on.</p>
<p class="p" dir="ltr">If <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/7043747123354" target="_blank">autoscoring</a> is turned on, bots are reviewed
automatically. However, you can also review your bots manually.</p>
<p class="p" dir="ltr"><strong class="ph b">To review a bot’s performance manually</strong></p>
<ol class="ol" id="topic_kyy_xvm_5hc__ol_lyy_xvm_5hc">
<li class="li" dir="ltr"><span class="ph">In <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/4408838272410" target="_blank">Quality assurance</a>, click <img class="image" id="topic_kyy_xvm_5hc__d8e80" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/qa_conversations_icon.png" width="22">
<strong class="ph b">Conversations</strong> in the sidebar.</span></li>
<li class="li" dir="ltr">Select an existing filter, or <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/7043759449114" target="_blank">create a new filter</a> to identify the bot conversations that you
want to review. For example, you might use any of the following filter conditions:<ul class="ul" id="topic_kyy_xvm_5hc__ul_myy_xvm_5hc">
<li class="li" dir="ltr"><strong class="ph b">Participant | is | <em class="ph i">&lt;name of your bot&gt;</em></strong></li>
<li class="li" dir="ltr"><strong class="ph b">Bot | is | <em class="ph i">&lt;name of your bot&gt;</em></strong></li>
<li class="li" dir="ltr"><strong class="ph b">Bot type | is | <em class="ph i">&lt;workflow or generative&gt;</em></strong></li>
<li class="li" dir="ltr"><strong class="ph b">Bot reply count | more than | 0</strong></li>
</ul>
<p class="p">Alternatively, <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/7043759586074" target="_blank">use a Spotlight filter</a> to find bot conversations.</p>
</li>
<li class="li" dir="ltr">Select the conversation you want to review.</li>
<li class="li" dir="ltr">In the <strong class="ph b">Review this conversation</strong> panel, select the bot you want to
review in the <strong class="ph b">Reviewee</strong> field, then select the <strong class="ph b">Scorecard</strong>.</li>
<li class="li" dir="ltr">Rate the bot’s performance for each category. See <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/7043669307418#topic_ows_lv2_p2c" target="_blank">Grading conversations</a>.<p class="p"><img class="image" id="topic_kyy_xvm_5hc__image_nyy_xvm_5hc" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/qa_bot_review.png" width="300"></p>
</li>
<li class="li" dir="ltr">(Optional) In the free-text field, enter comments about the bot’s
performance.</li>
<li class="li" dir="ltr"> Click <strong class="ph b">Submit</strong>.</li>
</ol>
<div class="p" dir="ltr">
<div class="note tip">
<span class="tiptitle">Tip:</span> Use the <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/7043724913690" target="_blank">Reviews dashboard</a> to analyze the results of a bot’s performance
evaluation. Use the <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/7418648572826" target="_blank">BotQA dashboard</a> to understand other important metrics about your
AI agent’s performance, including how often bot conversations were escalated to human
agents.</div>
</div>
</div>
</div>
<script id="docs-schema-markup" type="application/ld+json">{
"@context": "https://schema.org",
"@type": "TechArticle",
"@id": "https://support.zendesk.com/hc/en-us/articles/7418648293018-Evaluating-the-performance-of-AI-agents-using-Zendesk-QA",
"url": "https://support.zendesk.com/hc/en-us/articles/7418648293018-Evaluating-the-performance-of-AI-agents-using-Zendesk-QA",
"headline": "Evaluating the performance of AI agents using Zendesk QA",
"description": "You can evaluate the performance of AI agents in customer conversations by configuring review settings and using scorecards to manually or automatically assess their interactions.",
"articleSection": "Using AI in Zendesk QA",
"datePublished": "2024-06-12T17:53:26+00:00",
"dateModified": "2026-04-23T13:08:22.899351+00:00",
"inLanguage": "en-US",
"mainEntityOfPage": {
"@type": "WebPage",
"@id": "https://support.zendesk.com/hc/en-us/articles/7418648293018-Evaluating-the-performance-of-AI-agents-using-Zendesk-QA"
},
"author": {
"@type": "Organization",
"name": "Zendesk",
"url": "https://www.zendesk.com"
},
"publisher": {
"@type": "Organization",
"name": "Zendesk"
},
"isPartOf": {
"@type": "WebSite",
"name": "Zendesk Help",
"url": "https://support.zendesk.com/hc/en-us"
},
"about": {
"@type": "SoftwareApplication",
"name": "Zendesk"
}
}</script><script id="ai-summary-script">
//<![CDATA[
const ai_summary_style = document.createElement('style');
ai_summary_style.innerHTML = `#styled-article-body{padding:0 0 0}#article-body #docs-ai-summary{background-color:#f3f3f3;border:1px solid #d3d3d3;padding:10px 10px 6px;margin-bottom:16px}#article-body #docs-ai-summary #docs-ai-summary-toggle{font-size:.9em;font-weight:700;margin-bottom:4px;cursor:pointer;display:flex;justify-content:space-between;align-items:center}#article-body #docs-ai-summary #docs-ai-summary-toggle #up-triangle{margin-left:auto;padding-right:10px;display:none}#article-body #docs-ai-summary #docs-ai-summary-toggle #down-triangle{margin-left:auto;padding-right:10px}#article-body #docs-ai-summary #docs-ai-summary-content{font-size:.9em;padding-left:25px;padding-right:25px;margin-bottom:12px}`;
document.head.appendChild(ai_summary_style);

const toggleAISummary = () => {
const summary = document.getElementById("docs-ai-summary-content");
const upTriangle = document.getElementById("up-triangle");
const downTriangle = document.getElementById("down-triangle");
if (summary.style.display === "none") {
summary.style.display = "block";
upTriangle.style.display = "none";
downTriangle.style.display = "inline";
} else {
summary.style.display = "none";
upTriangle.style.display = "inline";
downTriangle.style.display = "none";
}
};

document.getElementById("docs-ai-summary-toggle").addEventListener("click", toggleAISummary);
//]]>
</script>
</body>