As datasets and models grow — especially with multimodal systems spanning text, images, video, and other unstructured data — ML teams inevitably hit a hard limit: your dataset or model no longer fits on a single GPU.

The decision is not whether to scale, but how to do so while continuing to use familiar frameworks like XGBoost, PyTorch, DeepSpeed, or Hugging Face Transformers.

Ray has emerged as the most widely adopted open-source framework for addressing this transition, with over 40k GitHub stars and more than 10 million weekly downloads. Teams at Uber, Discord, Attentive and many more use Ray to scale runs without rewriting the core training function logic.

• If you’re ready to learn Ray as a distributed AI framework, check out this free course.

If you’re still evaluating the move from single-node experiments to distributed training, the sections below outline the key technical and business considerations, and how Ray on Anyscale reduces both developer friction and operational risk.

LinkInfrastructure co-design is no longer optional

The traditional process where ML engineers train in a single-GPU and platform/MLOps teams “productionize” later breaks down in this new AI era as training itself is also an infrastructure problem.

Moving from a single GPU to multiple interconnected GPUs on one machine is usually manageable. Moving beyond that, across nodes, is where things get way harder. Multi-node training introduces failure modes that don’t exist in local workflows:

• GPU failures drop nodes and cause partial failures

• Dependency drift across machines

• Fragmented logs and metrics

• Debugging is no longer localized 

• Non-trivial integration with common ML frameworks

• Data that must be coordinated and moved across nodes

The getting started checklist quickly grows into a long set of undifferentiated infrastructure work:

• Provision and monitor multi-node GPU clusters

• Configure networking

• Wire up SSH launchers, Slurm or Kubernetes jobs

• Assign PyTorch DDP (Distributed Data Parallel) ranks to coordinate workers

• Manage data sharding and placement

• Building a fault-tolerant checkpointing system

The cost of a DIY approach isn’t just engineering time. It shows up as slower iteration, fragile experiments, missed deadlines, and models that never make it to production.

LinkData preparation starts to feel a lot more like training

As models and datasets become larger and more multimodal, the infrastructure boundary between data preparation and training starts to disappear. 

What were once ETL-like preprocessing steps are now batch inference steps with models e.g. using LLMs to generate labels, captions or running embedding generation at scale. This shifts data preparation from CPU-based ETL to GPU-centric batch inference. 

Optimizing infrastructure for training alone is no longer sufficient. Modern model development requires treating data processing and training as a single, end-to-end workflow, with shared scaling, scheduling, and failure-handling decisions.

LinkGPUs are scarce and expensive

Modern training runs span hours or days and often execute across multiple nodes. At that timescale, failures are expected, not exceptional. A single node reset can cost tens of thousands in compute, making reliability a core property of the training loop rather than an operational afterthought.

At the same time, GPUs can be hard to get access to. To manage limited capacity, teams frequently reserve large GPU allocations to handle peak demand across multiple workloads. In practice, higher-priority jobs like production inference often preempt long-running training runs. With a shared capacity model and hardware availability always shifting, training workloads must tolerate interruption and adapt to changing resources.

As a result, efficiency and adaptability become first-order concerns.

Checkpointing becomes essential, but coarse, epoch-level checkpoints are often not granular enough when epochs themselves take hours or days. High GPU utilization matters just as much as raw scale, which means teams need visibility into how resources are actually being used and the ability to optimize accordingly. And as hardware availability remains fluid, the ability to pause, resume, and adapt training runs to changing resource availability becomes critical rather than optional.

LinkWhy Ray — and Why Ray on Anyscale

When data preparation and training need to scale together, point solutions and frameworks break down. Ray provides a unified, distributed processing engine that accelerates end-to-end experimentation, often improving GPU utilization by 50–70%.

Ray handles the hard parts of distributed systems: 

• Worker coordination

• Scheduling

• Fault tolerance

• Resource management and more

With Ray Train library, developers express how the training function should scale in Python, while Ray handles the underlying parallelization. Teams can scale PyTorch, XGBoost, and Transformers without stitching together custom orchestration logic.

But Ray alone doesn’t remove the full operational burden. Teams still own cluster lifecycle management, environment consistency, failure recovery, and observability — work that slows experimentation and distracts from model development.

Ray on Anyscale builds on the same open-source runtime and closes this gap by adding purpose-built developer tooling (Anyscale Developer Central) that delivers:

• Interactive, multi-node development environments, accessible via VSCode, Cursor or Jupyter

• Centralized logs, metrics, and job state for debugging and optimizing distributed runs without SSH’ing into individual nodes

• Out-of-the-box dataset to model lineage

And a managed cluster control plane (Anyscale Cluster Controller) that has: 

• Automatic detection and replacement of unhealthy nodes to minimize manual intervention

• Fully managed cluster provisioning and teardown across training and data workloads

• Flexible deployment across cloud providers without changes to application code

• Built-in user and cost governance 

With Anyscale, distributed training stops feeling like a separate systems discipline. It becomes a natural way to scale with your models without turning ML development into an infrastructure maintenance job.

In short, Ray simplifies distributed execution. Anyscale turns it into a resilient platform you can rely on to drive developer velocity and infrastructure efficiency.

• Canva accelerated model development by 12x while reducing GPU costs by 50%

• Coinbase is able to process 50x larger datasets, while running 15x more jobs without costs increase

LinkLearn more

As multimodal datasets and model sizes grow, distributed training becomes the norm rather than the exception. 

Ray lowers the barrier to distributed training by letting ML engineers scale their existing code without becoming distributed systems experts. Ray on Anyscale completes the picture, pairing the Anyscale Runtime, powered by Ray, with production-grade cluster management and purpose-built developer tooling so teams can iterate faster and transition to production with confidence.

• Ready to learn Ray Train? Take our free course → Free Course

• Ready to get started with Anyscale? Experience Ray Train with our $100 new account credit → Free Trial