Evaluation Reviews add a powerful verification layer to your quality assurance process, ensuring every evaluation is accurate and consistent. This feature allows a second person (or a group of people) to review an evaluation before it's finalized and sent to an agent. It's perfect for spot-checking, building agent trust, and focusing manager attention on the most important interactions.
The Review Workflow
When an evaluation is flagged for review, it follows a simple, automated workflow designed to be seamless.
Evaluation Submitted: A standard evaluation is completed by your AI.
Review Triggered: A Scorecard Action you've created automatically flags the evaluation for review based on your specific criteria. At this point, the original evaluation is hidden from the agent, pending the review.
Evaluation Reviewed: The assigned reviewer is notified, and the task appears in their Work Queue on their homepage. The reviewer opens the evaluation, checks the scores and notes from the initial evaluation, edits them if needed, and then finalizes it by clicking "Mark as Reviewed."
Agent Receives Evaluation: Once the review is complete, the final, verified evaluation is released and becomes visible to the agent.
How to Set Up Evaluation Reviews
Instead of a single on/off switch, Evaluation Reviews are integrated directly into Scorecard Actions, giving you granular control over what gets reviewed and when.
Step 1: Configure Your Default Review Settings
To make setup easier, you can configure account-wide default settings for all your review actions.
Navigate to Settings > Reviews.
Here you can set the following defaults:
Default Assignment: Choose who gets assigned a review. This is limited to users with the ADMIN or EVALUATOR role.
Random Evaluator: Assigns the review to any active Admin or Evaluator. This is great for distributing the workload.
Select from specific reviewers: Define a specific user or group of users who will be assigned reviews. This is useful for routing reviews to a senior manager or team lead.
Default Deadline: Set a default timeframe for reviewers to complete their task. This helps ensure timely feedback. Options include 24 hours, 48 hours, 7 days, or no deadline.
These settings will be pre-populated whenever you create a new review Scorecard Action, but you can always override them for specific actions.
Step 2: Trigger Reviews with a Scorecard Action
Reviews are triggered using the Human Review action type within a Scorecard. This allows you to build a highly targeted and efficient review process.
When building a Scorecard, click to add an Action and select Require human review as the action type. You can trigger this action based on various conditions, such as:
The total score of an evaluation (e.g., trigger a review for all failed evaluations).
Every score submission.
The answer to a critical question (e.g., trigger a review if an agent failed a compliance question).
For a complete guide on setting up actions, please see our article on Scorecard Actions.
Managing Pending and Past-Due Reviews
You can easily track and manage all reviews assigned to you from your home page.
Notifications: Reviewers are notified of new assignments via an in-app notification and the task appearing in their Work Queue. You can also enable Daily Digests in your account settings to receive an email summary of all your to-do items, including pending reviews.
Past-Due Reviews: If a reviewer misses a deadline, the review is automatically marked as "Past Due" and moved to the top of their work queue for high visibility. Admins will also see all past-due items for the entire team in their daily digest email.
Reassigning Reviews: An Admin can reassign a pending review to another user at any time. This is useful if a reviewer is out of the office or overloaded.
Use Cases for Evaluation Reviews
Not sure where to start? Here are a few ways you can leverage Evaluation Reviews to improve your QA program:
Spot-Check Your AI: New to AI-powered QA? Build trust in the system by creating an action to have a human review evaluations completed by your AI. You can fine-tune scorecard questions and prompts based on the results.
Handle High-Stakes Interactions: Create an action on a compliance-heavy question. If an agent fails to give a proper disclosure, the evaluation can be automatically routed to a manager for immediate review and coaching.
Support New Agents: For agents in their first 30 days, you can create a dedicated scorecard that triggers a review for every single evaluation. This ensures they get detailed, verified feedback while they're still learning.
Train New Managers: Have a new team lead or manager? Assign them as the sole reviewer for their team's evaluations to get them up to speed on team performance.