Amazon SageMaker AI Announces Concurrency Sweeps for Generative AI Endpoints

Amazon SageMaker AI has announced a new feature called concurrency sweeps, designed to help users right-size their generative AI endpoints. Concurrency sweeps systematically benchmark the performance of endpoints by sending controlled levels of concurrent traffic and analyzing throughput and latency. This approach helps identify the ideal balance between cost and performance, ensuring that endpoints are neither over-provisioned nor under-provisioned.

According to the announcement, concurrency sweeps are built into Amazon SageMaker AI Inference Recommendations, eliminating the need for custom load-testing infrastructure. The feature allows users to deploy models, run automated sweeps, and use the results to make informed capacity decisions. For example, the maximum level of concurrency supported before latency becomes unacceptable is observed to be 80.

In the post, the team demonstrates how to deploy a model using the native vLLM container on Amazon SageMaker AI, run concurrency sweeps using the CreateAIBenchmarkJob API, and plot throughput against latency to identify the saturation point. Additionally, the max-concurrency-under-sla recipe is used to automatically discover the optimal concurrency for specific service level agreement (SLA) targets.

The complete notebook and further documentation on how to get started with your own models are available in the GitHub repository and the Generative AI Inference Recommendations documentation.

Source: aws-ml