Amazon SageMaker has launched the JumpStart Model Leaderboard, a tool that allows users to compare hundreds of open-weight foundation models directly in Amazon SageMaker Studio. The leaderboard provides quality scores from public benchmarks and performance and cost metrics, enabling users to make informed decisions.
What happened
Amazon SageMaker has introduced the JumpStart Model Leaderboard, a new feature that simplifies the process of comparing hundreds of open-weight foundation models available through JumpStart. The leaderboard offers quality scores from three public benchmarks (BIG-Bench Hard, GPQA Diamond, and MMLU-Pro) and performance and cost metrics measured by AWS. This allows users to make like-for-like comparisons without stitching together data from different sources. At launch, the leaderboard covers over 70 models from providers such as Meta, Google, Qwen, IBM, and Arcee. Users can filter models based on their requirements and deploy them with a few clicks.
What to weigh
- The leaderboard focuses on large language models initially.
- Users can filter and compare models based on their specific requirements.
FAQ
How can I access the leaderboard?
To access the leaderboard, open the Models page in SageMaker Studio and select the Leaderboard view.
What kind of benchmarks are included?
The leaderboard includes quality scores from three public benchmarks: BIG-Bench Hard, GPQA Diamond, and MMLU-Pro.
Can I compare performance and cost metrics?
Yes, the leaderboard provides performance and cost metrics measured by AWS using a standardized test workload.
Source: Amazon Web Services
