News

User Story

Engineering

Culture

Yi Cheng

Cade Daniel

Chen Shen

Liguang Xie

In this post, we discuss the importance of speed when loading large language models and what techniques we employed to make it 20x faster. In particular, we use the Llama 2 series of models. We share how you can reduce latency and costs using the Anyscale platform.

loading llms 20x faster

Loading Llama-2 70b 20x faster with Anyscale Endpoints

Eric Liang

Eric Liang headshot

Richard Liaw

Richard Liaw headshot

In this blog, we discuss continuous batching, a critical systems-level optimization that improves both throughput and latency under load for LLMs.

Cover image for 23X batch blog

Achieve 23x LLM Inference Throughput & Reduce p50 Latency

How continuous batching enables 23x throughput in LLM inference while reducing p50 latency

Jules S. Damji

Zhe Zhang

The Ray 2.3 release features exciting improvements across the Ray ecosystem. In this blog post, we will highlight new features, performance enhancements, and support for new platforms. 

main_release_image

Announcing Ray 2.3: performance improvements, new features and new platforms

Ray Summit 2024, the annual Ray community conference, is back in San Francisco, CA this September 30th through October 2nd. It is a two-day event with a third day dedicated to training.

Call for Proposals is open Blog Post

Ray Summit 2024 Call for Proposals is now open

Intel and Anyscale have integrated Ray with Gaudi3. Gaudi3 is AI-optimized for efficient training and other machine learning workloads.

Anyscale - Intel

Accelerating AI: Harnessing Intel(R) Gaudi(R) 3 with Ray 2.10

Anyscale is teaming with NVIDIA to combine the developer productivity of Ray Serve and RayLLM with the cutting-edge optimizations from NVIDIA Triton Inference Server software and the NVIDIA TensorRT-LLM library.

NVIDIA-Anyscale Logo 1280x680.jpg

Low-latency Generative AI Model Serving with Ray, NVIDIA Triton Inference Server, and NVIDIA TensorRT-LLM

Generating embeddings is a critical task for developing successful RAG applications, and Anyscalew with Pinecone is the most cost-efficient solution for this workflow.

blog-anyscale-pincone

RAG at Scale: 10x Cheaper Embedding Computations with Anyscale and Pinecone

This section is used to order the "Types" and "Tags" that show up for filters on the Blog Index

Types

Products / Libraries

Blog

Hosted Anyscale is in Private Preview. Click here to request access!

Anyscale is the leading AI application platform. With Anyscale, developers can build, run and scale AI applications instantly.

Posts by Cade Daniel

No posts found.