NVIDIA announced NeMo Compass and NeMo Eval Author, tools designed to help developers optimize AI agents by identifying recurring problems and creating custom evaluations. These tools are aimed at improving the reliability and performance of AI agents in production environments.
What happened
NVIDIA has introduced NeMo Compass and NeMo Eval Author to assist developers in optimizing AI agents. NeMo Compass analyzes stored agent traces and evaluation evidence to identify recurring problems and prioritize improvements. NeMo Eval Author provides coding agent skills and workflows to create custom evaluations that measure agent behavior against domain requirements. The complete optimization cycle involves updating the agent harness, such as modifying the Hermes SOUL.md system prompt, and rerunning evaluations on both development and held-out tasks to verify improvements and detect regressions. In a Dassault Systèmes engineering example, the same loop reduced malformed tool calls from 142 to zero and achieved a 28% relative improvement in overall evaluation score with 35% fewer tool calls per run across six scenarios.
What to weigh
- The optimization cycle requires updating the agent harness and rerunning evaluations.
- NeMo Eval Author supports two evaluation formats: Harbor and NVIDIA NeMo Gym.
FAQ
What does NeMo Compass do?
NeMo Compass analyzes stored agent traces and evaluation evidence to identify recurring problems, prioritize improvements, and link each finding to specific trace IDs.
What is NeMo Eval Author used for?
NeMo Eval Author provides coding agent skills and workflows to create custom evaluations in Harbor or NVIDIA NeMo Gym formats that measure agent behavior against domain requirements.
How does the optimization cycle work?
The complete optimization cycle updates the agent harness — such as modifying the Hermes SOUL.md system prompt — then reruns evaluations on both development and held-out tasks to verify improvements and detect regressions.
Source: NVIDIA