Last quarter, I led a small team updating our internal AI agent tool, which helps our customer support team draft responses to common inquiries. For months, we’d judge changes to our prompt workflows and retrieval logic by asking a handful of teammates to read sample answers and share their thoughts. But this approach was inconsistent: one colleague would praise a detailed, cited response for a password reset query, while another said it included too much extraneous policy language that slowed down frontline reps during peak support hours. We couldn’t reliably tell if a new release made answers better, or just shifted their tone or structure. We needed a way to turn subjective feedback into a measurable, repeatable metric.
The solution we landed on was a fixed evaluation set: a curated list of real user queries we’d collected over several months, each paired with a reference answer that our team had agreed represented a strong, ideal response. To automate the repetitive work of running queries and compiling scores, we integrated with the FastGPT open-source platform. The platform let us configure a dedicated evaluation workflow, upload our query-reference pairs, and connect the workflow to our staging agent endpoints. We didn’t have to build a full evaluation framework from scratch; we just tweaked a few pre-built prompt templates to match our team’s scoring rubric—focused on relevance, factual accuracy, and clarity—and set the workflow to run automatically every time we pushed a new deployment. This eliminated the need to coordinate ad-hoc feedback sessions for every minor change, streamlining our release prep process.
Our first test with this setup came two weeks later, when we updated our context retrieval logic to prioritize more recent support tickets. Before rolling the change out widely, we ran the fixed evaluation set. The average score across all queries was higher than the previous release’s average, with consistent improvements across every high-volume query category, so we felt confident the change improved answer quality without needing to gather fresh team feedback. We’ve since added this evaluation step to every minor and major release cycle. It doesn’t replace real user feedback from support tickets, but it gives us a consistent, apples-to-apples comparison between two versions of our agent.
Before committing any agent update to production, we use a short verification checklist to ensure our evaluation workflow runs effectively:
- Confirm the fixed evaluation set covers all active, high-volume support query categories
- Verify the scoring rubric aligns with current team standards for response quality
- Cross-check a random sample of automated scores against human reviewer feedback
- Ensure any recently added agent features are represented in the evaluation queries
We encountered one key failure mode early on, when we updated our agent to include direct links to internal knowledge base articles in responses. After deploying the change, the automated evaluation scores remained high, but frontline support reps flagged that responses for queries about accessing the knowledge base didn’t include those critical links. This gap arose because none of the queries in our fixed evaluation set were tied to that specific request type, so the automated scoring didn’t account for the new feature’s requirements. We then updated our evaluation set to include queries covering that use case, ensuring future releases would be scored against relevant, up-to-date criteria.
There are important caveats to this approach, though. First, our scoring rubric is tailored to our specific use case: customer support responses. It wouldn’t work directly for other use cases like technical documentation or creative writing without adjusting both the reference answers and the scoring criteria. For example, a technical documentation agent would need reference answers focused on step-by-step instructions rather than conversational support tone, and a scoring rubric prioritizing technical correctness over approachability. Second, the automated scoring we used still relies on an LLM grader, which can have its own blind spots or biases. We make a point to cross-check a small random sample of scores each month to ensure they align with what a human reviewer would assign; sometimes the grader might misinterpret industry-specific jargon our support team uses regularly, or mark a response as low relevance if it includes an internal link the grader doesn’t recognize as part of our internal tools. Finally, this setup only measures answer quality, not other critical factors like inference speed or operational costs, which we track separately using internal monitoring tools, since our primary goal for this evaluation workflow is to standardize output quality across releases.