All products

Evaluate AI models with sudoku

Compare models on the same generated puzzles and inspect exactly which answer broke a grid rule.

Evaluate AI models with sudoku when you need a small comparison where correctness has a clear definition. A complete-looking answer may have replaced a given digit in the corner. The puzzletide-agent-evals workflow gives each generated task an ID and checks the returned grid against the original clues and the rules. You can keep the failing examples alongside the overall score.

There is a second task family for word searches. Here the response supplies start and end coordinates, so the grader can read the letters along the proposed line. Keep the same puzzle type, difficulty, task count and seed when comparing models. Record which model produced each answer file yourself.

What it does

  • Generate synthetic tasks with stable IDs.
  • Give two models the same instructions and puzzle payloads.
  • Check a completed sudoku against its given cells and unit constraints.
  • Grade word-search answers by their reported coordinates.
  • Keep per-task reasons with the pass-rate summary.

Install and use

  1. Use Node.js 18.17 or later and PuzzleTide’s official CLI. You arrange access to the model being tested.
  2. Generate a fixed set, then collect each model’s responses under the matching task IDs.
  3. Run the local grader and review a failed item alongside the summary.

FAQ

Source

Made by Caravaca Labs. Original skill: puzzletide-agent-evals, published on GitHub under the MIT licence.

View on GitHub