News

User Story

Engineering

Culture

Yi Cheng

Cade Daniel

Chen Shen

Liguang Xie

In this post, we discuss the importance of speed when loading large language models and what techniques we employed to make it 20x faster. In particular, we use the Llama 2 series of models. We share how you can reduce latency and costs using the Anyscale platform.

loading llms 20x faster

Loading Llama-2 70b 20x faster with Anyscale Endpoints

image3

​​Reinventing Multi-Modal Search with Anyscale and MongoDB

Solve resource fragmentation with Anyscale's new Replica Compaction feature.  Easily optimize resource usage for online inference replication.

Anyscale’s New Replica Compaction

Introducing Anyscale’s New Replica Compaction

Deploy Ray Serve with up to 50% fewer nodes using Anyscale Replica Compaction

Anyscale - LLM Router

Building an LLM Router for High-Quality and Cost-Effective Responses

Ray Spotlight

Ray Spotlight: How we delivered Ray weekly releases

Execute end-to-end LLM workflows to develop & productionize LLMs at scale.

End-to-end LLM Guide header

End-to-end LLM Workflows Guide

This section is used to order the "Types" and "Tags" that show up for filters on the Blog Index

Types

Products / Libraries

Blog

Anyscale Preview is now available! Login today to get free $25 in hosted Anyscale credits 🚀

Anyscale is the leading AI application platform. With Anyscale, developers can build, run and scale AI applications instantly.

Anyscale

Posts by Yi Cheng

No posts found.