All products

Evaluate Omni AI answers with a test set

Save representative analytics questions, run them against a model or branch and inspect the judge's verdict. Reuse the same set after a change.

Evaluate Omni AI answers with a test set before promoting a change to the company model. A new definition of revenue might fix sales questions but change how returns are counted. Keeping both kinds of question in the set gives reviewers something concrete to compare.

The omni-ai-eval skill uses Omni's hosted evaluation system. Its accuracy judge can consider topic selection, the query, its results and the answer delivered to the user. An optional reference expectation helps make a business convention explicit when the question alone leaves room for interpretation.

What it does

  • Create a reusable set of team questions
  • Add an expectation for a business-specific definition
  • Run the same set against the main model and a branch
  • Inspect the rationale and confidence for each verdict
  • Repeat an evaluation after revising model context

Install and use

  1. Install the skill and official Omni CLI. Use an authenticated profile with evaluation access to the shared model, an AI-ready topic and any required AI credits.
  2. Ask, “Compare these weekly returns questions on main and our proposed branch. Flag changes in date handling and show the judge's reasoning.”
  3. Review the results against the business definition. Use only prompts and expected answers that may be stored in the hosted system.

FAQ

Source

Made by Omni Analytics, Inc. Original skill: omni-ai-eval, published on GitHub under the Apache-2.0 licence.

View on GitHub