News

User Story

Engineering

Culture

Yi Cheng

Cade Daniel

Chen Shen

Liguang Xie

In this post, we discuss the importance of speed when loading large language models and what techniques we employed to make it 20x faster. In particular, we use the Llama 2 series of models. We share how you can reduce latency and costs using the Anyscale platform.

loading llms 20x faster

Loading Llama-2 70b 20x faster with Anyscale Endpoints

Eric Liang

Eric Liang headshot

Richard Liaw

Richard Liaw headshot

In this blog, we discuss continuous batching, a critical systems-level optimization that improves both throughput and latency under load for LLMs.

Cover image for 23X batch blog

Achieve 23x LLM Inference Throughput & Reduce p50 Latency

How continuous batching enables 23x throughput in LLM inference while reducing p50 latency

Clark Zinzow

Alex Wu

Jiajun Yao

blog-ray-datasets-thumb

Ray Datasets for large-scale machine learning ingest and scoring

Waleed Kadous

3rdGenTasks andActors

Data Ingest in a Third-Generation ML Architecture | Anyscale

Deep Dive: Data Ingest in a Third Generation ML Architecture

The Ray Summit 2025 CfP is now open for submissions. The premier AI conference will be held November 3-5 in San Francisco, and the due date for CfPs is June 30th, 2025.  

Ray Summit 2025 CFP Open

Ray Summit 2025: Call for Proposals Closes on June 30th, 2025

Google + Anyscale

Google Cloud Integrates Anyscale's RayTurbo with GKE

Simplifying AI Development at Scale: Google Cloud Integrates Anyscale's RayTurbo with GKE

At Ray Summit 2024, we brought together AI experts and practitioners to dive into the future of distributed AI and scalable machine learning.

image3

Ray Summit 2024: Breaking Through the AI Complexity Wall

This section is used to order the "Types" and "Tags" that show up for filters on the Blog Index

Types

Products / Libraries

Blog

Ray Summit 2025 Call for Speakers is Now Open!

Anyscale is the leading AI application platform. With Anyscale, developers can build, run and scale AI applications instantly.

Anyscale