Creating AI prompt-based rating categories

<body>
<p id="docs-hc-snippet" style="display: none;">In Zendesk QA, whilespotlight insightsflag tickets,rating categoriesscore agents.</p>
<div class="article-banners">
<div id="docs-wmp-link">
<a href="https://support.zendesk.com/hc/en-us/articles/5411234991258-plan" rel="noopener" target="_blank">What's my plan?</a>
</div>

<div class="article-banner" id="addon_zqa">
<table>
<tbody>
<tr>
<td>
<strong>Add-on</strong>
</td>
<td>Quality Assurance (QA) or Workforce Engagement Management (WEM)</td>
</tr>
</tbody>
</table>
</div>

</div>
<div id="docs-ai-summary">
<p id="docs-ai-summary-toggle">Summary: <span id="up-triangle">◀</span><span id="down-triangle">▼</span></p>
<p id="docs-ai-summary-content">Create AI prompt-based rating categories to score agent conversations with natural language prompts, custom templates, and rating scales. You can test and refine prompts, use AI suggestions or validation, and track prompt history before adding the category to scorecards. Up to 10 active AI prompt-based categories and spotlight insights are allowed per account.</p>
</div>

<div class="body">

<div class="p">
<div class="note fastpath">
<span class="fastpathtitle">Location: </span> Zendesk QA &gt; Settings &gt; AI</div>
</div>
<p class="p" dir="ltr">In Zendesk QA, while <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/7043759586074" target="_blank">spotlight insights</a> flag tickets, <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/7043712922522" target="_blank">rating categories</a> score agents.</p>
<p class="p" dir="ltr">In addition to the <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/7043747123354#understanding_autoscoring_categories" target="_blank">system categories</a>, <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/10301342601626" target="_blank">manual categories</a>, and <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/9302967236890" target="_blank">exact text-match categories</a>, admins and workspace managers can create
AI prompt-based rating categories. </p>
<p class="p" dir="ltr">All rating categories allow you to effectively assess customer conversations.
However, by using AI prompt-based categories, you can ask targeted questions about
conversations using natural language and automatically score them based on your criteria.</p>
<p class="p" dir="ltr">Zendesk QA prompt-based AI insights leverage the latest AI models, letting you
create custom AI rating categories from scratch or choose from our ready-to-use <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/9250434748058#topic_hws_czb_yfc" target="_blank">AI prompt templates</a>. The AI Prompt category criteria
define the agent's behavior.</p>
<div class="p">
<div class="note tip">
<span class="tiptitle">Tip:</span> To ensure proper functionality, both the <em class="ph i">Automatic scoring with
AutoQA</em> and the <em class="ph i">LLM-based AutoQA </em><a class="xref" href="https://support.zendesk.com/hc/en-us/articles/7043669430426#topic_wsw_cly_ydc" target="_blank">account settings</a> must be turned on.</div>
</div>
<p class="p" dir="ltr">Related articles</p>
<ul class="ul" id="topic-1__ul_nc4_b2c_yfc">
<li class="li"><a class="xref" href="https://support.zendesk.com/hc/en-us/articles/9224552305946" target="_blank">About AI insights in Zendesk QA</a></li>
<li class="li" dir="ltr"><a class="xref" href="https://support.zendesk.com/hc/en-us/articles/9464975500954" target="_blank">Best practices for creating AI insights prompts in Zendesk
QA</a></li>
<li class="li" dir="ltr"><a class="xref" href="https://support.zendesk.com/hc/en-us/articles/9327313916570" target="_blank">Creating AI prompt-based spotlight insights</a></li>
</ul>
<p class="p" dir="ltr">Admins and workspace managers can have up to 10 active AI prompt-based rating
categories and AI prompt-based spotlight insights per account. When you reach this limit, you
must either mark one as inactive or delete it before you can create or activate more.</p>
<p class="p" dir="ltr"><strong class="ph b">To create a custom AI prompt-based rating category</strong></p>
<p class="p"><img class="image img-screenshot" id="topic-1__image_g1x_3hg_bkc" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/qa_ai_prompt_category.png" width="850"></p>
<div class="p">
<ol class="ol" id="topic-1__ol_f33_jsc_yfc">
<li class="li" dir="ltr"><span class="ph">In <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/4408838272410" target="_blank">Quality assurance</a>, click your profile icon in the top-right
corner.</span></li>
<li class="li" dir="ltr"><span class="ph">Click <strong class="ph b">Settings</strong>.</span></li>
<li class="li" dir="ltr"><span class="ph">In the sidebar under <strong class="ph b">Account</strong>, click <strong class="ph b">AI</strong>.</span></li>
<li class="li">Click <strong class="ph b">Create AI insight</strong>.</li>
<li class="li">Select the <strong class="ph b">Prompt-based</strong> AI insight type.</li>
<li class="li">(Optional) Click <strong class="ph b">Prompt templates</strong> to select a template from our ready-to-use
<a class="xref" href="https://support.zendesk.com/hc/en-us/articles/9250434748058" target="_blank">AI prompt templates</a>, then click <strong class="ph b">Use template</strong>.</li>
<li class="li">Enter an <strong class="ph b">AI insight name</strong>. </li>
<li class="li">Under <strong class="ph b">Evaluation filters</strong>, select the type of participants you want to include in
the conversations to analyze, such as human agents and AI agents, the channels, tags, and
source types. See <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/7043759449114" target="_blank">Understanding conversation filter types in Zendesk
QA</a>.</li>
<li class="li">Customize any template to fit your specific needs or create a new prompt from
scratch:<ol class="ol" id="topic-1__ol_ihn_gvc_yfc" type="a">
<li class="li" dir="ltr">In <strong class="ph b">What do you want to detect?</strong>, clearly describe what this insight
should identify and accomplish. <p class="p">Specify that the focus is on the behavior of agents
and use objective, measurable language. You can enter up to 2,000 characters.
</p>
<p class="p">Be sure to follow our best practices for <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/9464975500954#topic_jqy_st2_m2c" target="_blank">writing prompts for AI insights</a> to ensure
your insights are accurate and actionable.</p>
</li>
<li class="li">Select a <strong class="ph b">Rating scale</strong>.<p class="p">Smaller scales work best for quick, consistent
reviews across a large volume of tickets, while larger scales provide nuance and are
better suited for coaching and improving quality. See <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/7043701093786#category_scores" target="_blank">Understanding how category scores are
calculated</a>.</p>
<p class="p">The following rating scales are available:</p>
<div class="note tip">
<span class="tiptitle">Tip:</span> You can <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/8875998154906#topic_ufd_t3n_zfc" target="_blank">edit your account’s scorecard settings</a> to
use numbers instead of emojis.</div>
<ol class="ol" id="topic-1__ol_l1f_vyf_53c" type="i">
<li class="li" dir="ltr">
<strong class="ph b">Binary</strong>: A binary scale is the fastest and most consistent scale
because it has only two options: <em class="ph i">good</em> and <em class="ph i">bad</em>. It's ideal for large
volumes where speed is important.</li>
<li class="li" dir="ltr">
<strong class="ph b">3-point</strong>: A 3-point scale can still be graded quickly but adds a
little more nuance by including a middle option of <em class="ph i">satisfactory</em> in addition
to <em class="ph i">good</em> and <em class="ph i">bad</em>.</li>
<li class="li" dir="ltr">
<strong class="ph b">4-point</strong>: A 4-point scale encourages critical thinking by
requiring a decisive assessment of <em class="ph i">good</em>, <em class="ph i">slightly good</em>, <em class="ph i">slightly
bad</em>, or <em class="ph i">bad</em> without a neutral option.</li>
<li class="li" dir="ltr">
<strong class="ph b">5-point</strong>: A 5-point scale offers more detailed feedback and
resembles the academic grading system (A = 5, B = 4, and so on). The drawback of
this type of scale is that grading can often take longer, since there is more for
your reviewers to consider.</li>
</ol>
</li>
</ol>
<div class="note note">
<span class="notetitle">Note:</span> If AI can't find a clear answer based on the prompts, the category is
automatically <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/7043701093786#topic_y5g_2cj_4fc" target="_blank">rated as N/A</a>.</div>
</li>
<li class="li">(Optional) If the <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/7043669430426#topic_wsw_cly_ydc" target="_blank">AI suggestions toggle is turned on in your account
settings</a>, click <strong class="ph b">Enhance</strong>. The system analyzes your prompt to resolve
ambiguity, vagueness, or poor formatting. It also refines your core intent by adding the
following details:<ol class="ol" id="topic-1__ol_o2d_nns_p3c" type="a">
<li class="li" dir="ltr">The main ask of your prompt</li>
<li class="li" dir="ltr">What it is</li>
<li class="li" dir="ltr">What it is not</li>
<li class="li" dir="ltr">You prompt answer description</li>
<li class="li" dir="ltr">An example of a negative prompt score</li>
<li class="li" dir="ltr">An example of a positive prompt score</li>
<li class="li" dir="ltr">What needs clarifying</li>
<li class="li" dir="ltr">Your prompt answer options</li>
</ol>
<div class="p">
<div class="note tip">
<span class="tiptitle">Tip:</span> Evaluation typically uses conversation data only. However, when
metadata elements appear in the prompt, such as response time, tags, channel, and
internal notes, they’re also taken into account during prompt evaluation. See <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/9464975500954#topic_sbq_lw4_jhc" target="_blank">Writing prompts with metadata</a>. </div>
</div>
</li>
<li class="li">(Optional) Click <strong class="ph b">Review the enhanced prompt </strong>to see what changed from your
original prompt and access detailed suggestions.</li>
<li class="li">(Optional) Click <strong class="ph b">Use original prompt </strong>to revert to your original prompt.</li>
<li class="li">(Optional) If the <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/7043669430426-Viewing-and-updating-Zendesk-QA-account-settings#topic_wsw_cly_ydc" target="_blank">AI suggestions toggle is turned off in your account
settings</a>, click <strong class="ph b">Validate</strong> to receive feedback and tips on how to improve
your prompt without changing it.</li>
<li class="li">Under <strong class="ph b">Suggested usage</strong>, select <strong class="ph b">Use as a category on a scorecard</strong>.</li>
<li class="li">(Optional) Click <strong class="ph b">Test prompt </strong>to confirm that your prompt works correctly and
meets the specified criteria.<p class="p" dir="ltr"><a class="xref" href="https://support.zendesk.com/hc/en-us/articles/7043759449114" target="_blank">A conversation filter view</a> shows the total number of
conversations found based on the prompted insights.</p>
<p class="p"><img class="image img-screenshot" id="topic-1__image_plw_xxd_tgc" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/ai_prompt_select_for_testing.png" width="800"></p>
<p class="p">To keep the
pool of conversations for testing specific, the filter displays a list of closed
conversations created in the last seven days that have more than four replies, by
default, with a limit of 100 conversations. </p>
</li>
<li class="li">(Optional) Click <strong class="ph b">Select for testing</strong>. The conversations found using this prompt
are listed in the right-side panel, displayed 10 at a time.<p class="p"><img class="image img-screenshot" id="topic-1__image_j5b_bzd_tgc" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/qa_Manual+vs+Auto+Review_category_prompt.png" width="800"></p>
</li>
<li class="li">(Optional) If you’ve validated and tested this prompt, then edited it and are testing it
again, a prompt history is displayed. This lets you view earlier versions of the prompt
and the match rate, which indicates the results evaluated based on that version. Check how
often auto review and manual review scores align to get a clearer overview of your prompt
performance. You can use this history to select the most suitable prompt version before
moving on to its final creation. <p class="p"><img class="image img-screenshot" id="topic-1__image_xdl_5xg_23c" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/qa_prompt_history_category.png" width="450"></p>
</li>
<li class="li">(Optional) Test your results:<ol class="ol" id="topic-1__ol_l4l_mzd_tgc" type="a">
<li class="li">(Optional) Click <strong class="ph b">Review </strong>next to each conversation and manually score them
based on the prompt you are creating so you can compare the results and see how often
auto review and manual review scores align. </li>
<li class="li">Click <strong class="ph b">Run auto review</strong> to see the results of your prompt conditions
automatically applied to all listed conversations under <strong class="ph b">Auto</strong>. <p class="p" dir="ltr">If
you see many conversations highlighted in yellow, this likely indicates a
misalignment between how you want to score conversations and how AI is scoring them.
You should consider rephrasing your prompt.</p>
<div class="note attention">
<span class="attentiontitle"></span> When
rephrasing what you want to detect with your prompt, if your changes are minor, you
can click <strong class="ph b">Run auto review</strong> again. However, if your changes are more
significant, such as modifying it to evaluate the customer instead of the agent, the
prompt type might change from category to spotlight. In this case, you'll need to
return to step 14 and click <strong class="ph b">Test prompt </strong>again.</div>
</li>
</ol>
</li>
<li class="li">Click<strong class="ph b"> Create category</strong>. </li>
<li class="li">After creating it, you must <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/7043760215194" target="_blank">add the category to your scorecards</a> to start using it. </li>
</ol>
</div>
</div>
<script id="docs-schema-markup" type="application/ld+json">{
"@context": "https://schema.org",
"@type": "TechArticle",
"@id": "https://support.zendesk.com/hc/en-us/articles/9277382490138",
"url": "https://support.zendesk.com/hc/en-us/articles/9277382490138",
"headline": "Creating AI prompt-based rating categories",
"description": "You can create AI prompt-based rating categories to score agent conversations with natural language criteria, test prompts, and add the category to scorecards.",
"articleSection": "Using AI in Zendesk QA",
"datePublished": "2025-05-20T13:09:49+00:00",
"dateModified": "2026-08-03T09:52:18.522431+00:00",
"inLanguage": "en",
"mainEntityOfPage": {
"@type": "WebPage",
"@id": "https://support.zendesk.com/hc/en-us/articles/9277382490138"
},
"author": {
"@type": "Organization",
"name": "Zendesk",
"@id": "https://www.zendesk.com"
},
"publisher": {
"@type": "Organization",
"name": "Zendesk",
"@id": "https://www.zendesk.com"
},
"isPartOf": {
"@type": "WebSite",
"name": "Zendesk Help",
"@id": "https://support.zendesk.com/hc/"
},
"about": {
"@type": "SoftwareApplication",
"name": "Zendesk",
"@id": "https://www.zendesk.com",
"softwareAddOn": {
"@type": "SoftwareApplication",
"name": "Quality Assurance",
"@id": "https://support.zendesk.com/hc/en-us/sections/6999625340058"
}
}
}</script><script id="fast-path-script">
//<![CDATA[
const fast_path_style = document.createElement('style');
fast_path_style.innerHTML = `#article-body .fastpath{font-size:.9em;background-color:#f3f3f3;border:1px solid #d3d3d3;padding:10px 10px 8px;margin-bottom:16px}#article-body .fastpath::before{content:none!important}`;
document.head.appendChild(fast_path_style);
//]]>
</script><script id="ai-summary-script">
//<![CDATA[
const ai_summary_style = document.createElement('style');
ai_summary_style.innerHTML = `#styled-article-body{padding:0 0 0}#article-body #docs-ai-summary{background-color:#f3f3f3;border:1px solid #d3d3d3;padding:10px 10px 6px;margin-bottom:16px}#article-body #docs-ai-summary #docs-ai-summary-toggle{font-size:.9em;font-weight:700;margin-bottom:4px;cursor:pointer;display:flex;justify-content:space-between;align-items:center}#article-body #docs-ai-summary #docs-ai-summary-toggle #up-triangle{margin-left:auto;padding-right:10px;display:none}#article-body #docs-ai-summary #docs-ai-summary-toggle #down-triangle{margin-left:auto;padding-right:10px}#article-body #docs-ai-summary #docs-ai-summary-content{font-size:.9em;padding-left:25px;padding-right:25px;margin-bottom:12px}`;
document.head.appendChild(ai_summary_style);

const toggleAISummary = () => {
const summary = document.getElementById("docs-ai-summary-content");
const upTriangle = document.getElementById("up-triangle");
const downTriangle = document.getElementById("down-triangle");
if (summary.style.display === "none") {
summary.style.display = "block";
upTriangle.style.display = "none";
downTriangle.style.display = "inline";
} else {
summary.style.display = "none";
upTriangle.style.display = "inline";
downTriangle.style.display = "none";
}
};

document.getElementById("docs-ai-summary-toggle").addEventListener("click", toggleAISummary);
//]]>
</script>
</body>