News

User Story

Engineering

Culture

Stephanie Wang

Scott Lee

Cheng Su

Hao Chen

Eric Liang

Eric Liang headshot

Data loading & preprocessing can easily become the performance bottleneck in ML pipelines. With Ray Data, data loading can be fast, flexible and scalable.

featured image ml training with ray data

Fast, Flexible and Scalable Data Loading for ML Training

Fast, flexible, and scalable data loading for ML training with Ray Data

Cade Daniel

Chen Shen

Richard Liaw

Richard Liaw headshot

In this blog, we discuss continuous batching, a critical systems-level optimization that improves both throughput and latency under load for LLMs.

Cover image for 23X batch blog

Achieve 23x LLM Inference Throughput & Reduce p50 Latency

How continuous batching enables 23x throughput in LLM inference while reducing p50 latency

This blog post delves into how Ray Data streaming works and how to use it for your own ML pipelines distributed across both CPU and GPU devices.

main_streaming_image

How to Stream Distributed Execution Across CPUs & GPUs

Streaming distributed execution across CPUs and GPUs

Amog Kamsetty

Jules S. Damji

We conduct a comparison of three different solutions for offline batch inference: AWS SageMaker Batch Transform, Apache Spark, and Ray Data.

Figure 1. Throughput difference among three different solutions for 10GB batch inference


Figure 1

Offline Batch Inference | Ray, Apache Spark & SageMaker

Offline Batch Inference: Comparing Ray, Apache Spark, and SageMaker

Antoni Baum

Jun Gong

Kai Fricke

Learn the infrastructure challenges for supporting workloads production deployments around foundation models, and how Ray tackles these challenges.

layered-stack

How Ray Solves Generative AI & LLM Infrastructure Challenges

How Ray solves common production challenges for generative AI infrastructure

Robert Nishihara

More companies are needing to train & deploy many small ML models, often hundreds or thousands. Learn how Ray and Anyscale make that possible.

Many_model_image

Training 1 Million ML Models in Record Time | Anyscale

Training One Million Machine Learning Models in Record Time with Ray

Clark Zinzow

Alex Wu

Jiajun Yao

blog-ray-datasets-thumb

Ray Datasets for large-scale machine learning ingest and scoring

Waleed Kadous

3rdGenTasks andActors

Data Ingest in a Third-Generation ML Architecture | Anyscale

Deep Dive: Data Ingest in a Third Generation ML Architecture

Zhe Zhang

Ray Distributed Library Patterns (Figure 2)

Ray Distributed Library Patterns

Sang Cho

AspectDataProcessingImage

Data Processing Support in Ray

Ben Lorica

Ion Stoica

Ion Stoica headshot

Birds

The Ideal Foundation for a General Purpose Serverless Platform

ray-1.0 image

Announcing Ray 1.0

Ray Summit 2024, the annual Ray community conference, is back in San Francisco, CA this September 30th through October 2nd. It is a two-day event with a third day dedicated to training.

Call for Proposals is open Blog Post

Ray Summit 2024 Call for Proposals is now open

Intel and Anyscale have integrated Ray with Gaudi3. Gaudi3 is AI-optimized for efficient training and other machine learning workloads.

Anyscale - Intel

Accelerating AI: Harnessing Intel(R) Gaudi(R) 3 with Ray 2.10

Anyscale is teaming with NVIDIA to combine the developer productivity of Ray Serve and RayLLM with the cutting-edge optimizations from NVIDIA Triton Inference Server software and the NVIDIA TensorRT-LLM library.

NVIDIA-Anyscale Logo 1280x680.jpg

Low-latency Generative AI Model Serving with Ray, NVIDIA Triton Inference Server, and NVIDIA TensorRT-LLM

Generating embeddings is a critical task for developing successful RAG applications, and Anyscalew with Pinecone is the most cost-efficient solution for this workflow.

blog-anyscale-pincone

RAG at Scale: 10x Cheaper Embedding Computations with Anyscale and Pinecone

This section is used to order the "Types" and "Tags" that show up for filters on the Blog Index

Types

Products / Libraries

Blog

Hosted Anyscale is in Private Preview. Click here to request access!

Anyscale is the leading AI application platform. With Anyscale, developers can build, run and scale AI applications instantly.

Anyscale

Posts by Eric Liang

No posts found.