Skip to content

Amazon Launches SageMaker JumpStart Model Leaderboard

Amazon SageMaker introduces a new Model Leaderboard to help users compare hundreds of open-weight foundation models on quality, speed, and cost.

Sources in the coveragePerformance and quality statements from a vendor are claims unless the coverage cites independent measurements.

Amazon SageMaker has launched the JumpStart Model Leaderboard, a tool that allows users to compare hundreds of open-weight foundation models directly in Amazon SageMaker Studio. The leaderboard provides quality scores from public benchmarks and performance and cost metrics, enabling users to make informed decisions.

What happened

Amazon SageMaker has introduced the JumpStart Model Leaderboard, a new feature that simplifies the process of comparing hundreds of open-weight foundation models available through JumpStart. The leaderboard offers quality scores from three public benchmarks (BIG-Bench Hard, GPQA Diamond, and MMLU-Pro) and performance and cost metrics measured by AWS. This allows users to make like-for-like comparisons without stitching together data from different sources. At launch, the leaderboard covers over 70 models from providers such as Meta, Google, Qwen, IBM, and Arcee. Users can filter models based on their requirements and deploy them with a few clicks.

What to weigh

  • The leaderboard focuses on large language models initially.
  • Users can filter and compare models based on their specific requirements.

FAQ

How can I access the leaderboard?

To access the leaderboard, open the Models page in SageMaker Studio and select the Leaderboard view.

What kind of benchmarks are included?

The leaderboard includes quality scores from three public benchmarks: BIG-Bench Hard, GPQA Diamond, and MMLU-Pro.

Can I compare performance and cost metrics?

Yes, the leaderboard provides performance and cost metrics measured by AWS using a standardized test workload.


Source: Amazon Web Services

Keep following technology

Continue from here.

Explore more coverage or subscribe to Fathom Layer updates.

Back to the news desk ↗

Occasional verified technology updates. Unsubscribe anytime. See our privacy policy.