Chef Robotics scales autonomous food assembly robots on Anyscale

Chef Robotics builds autonomous robotic systems for food manufacturing facilities, where robots pick ingredients from source containers, portion them, and place them into their final position in a prepackaged meal. The work depends on machine learning rather than hand coded rules, since customers judge output on qualities like weight consistency, appearance, and spillage that are difficult to define with fixed metrics across the wide range of ingredients Chef's robots handle.

Now Chef is doing research to expand into lower-volume operations like fast-casual restaurants and ghost kitchens. The approach above extends into more general purpose systems as well, including a recent bimanual robotic platform that Chef built to serve food businesses where flexibility and dexterity matter most, and throughput requirements are much lower than high-volume manufacturing. These include ghost kitchens, fast casuals, QSRs, prisons, hotels, stadiums, event venues, corporations, K-12, and universities. This is where Chef is able to leverage pretrained frontier-grade foundation models as a base and post-train them with foodstuff data for food manipulation.

To train the foundation models that give Chef's robots this kind of adaptability, the team needed a distributed compute platform that could keep pace with an experimental, fast moving research process and hold up as those workloads grew into larger, multi GPU training runs. Chef's applied AI team, working closely with the autonomy team that owns production deployment, adopted Ray on Anyscale to run that training.

LinkChallenges

As Chef pushed to train larger and more capable foundation models, the team needed infrastructure that could keep up with an experimental research process, support training runs that were quickly outgrowing a single machine, and do both without adding cost or engineering overhead the applied AI team could not absorb. 

Three challenges stood in the way:

• Iteration speed needed to match an unpredictable research and data collection cycle. Chef's data collection for new models is ad hoc rather than scheduled, so once a new batch of multi camera video and sensor data was ready, the team needed to start training on it right away. Training runs for the foundation models, including the Pi0 based models Chef trains for food manipulation tasks that are too complex to code by hand, took long enough that they slowed down how many ideas the team could test in a given week, and the workflow of manually provisioning GCP instances and managing training scripts added friction on top of that.

• Distributed training reliability needed to hold up as GPU jobs scaled beyond a single node. Early on, losing access to a cluster carried real risk, since regaining H100 capacity could take days once it was gone, and GPU availability could shift noticeably from one week to the next or even one day to the next. As training moved from single node runs toward larger multi GPU clusters, Chef needed a way to secure GPU capacity reliably and scale training across nodes without that scaling introducing new points of failure.

• Cost efficiency needed to scale without giving up flexibility. Running large training jobs on GCP directly meant absorbing the operational cost of managing spot instance interruptions by hand, including job crashes and manual restarts that could set a run back significantly. Chef wanted the cost advantages of spot capacity where it made sense, without asking a lean applied AI team to build and maintain that reliability layer themselves.

"Our customers care about the overall quality of the deposit: the weight consistency, how it looks, and whether anything spills. There is no obvious metric for spillage or for how well something is deposited, which is part of what makes this such a hard problem to solve without a learned approach."

Luis Rayas | CTO, Chef Robotics

LinkThe Solution

Chef's autonomy team first evaluated Ray through Anyscale directly, and an early proof of concept made the case for adopting it as the training platform for Chef's applied AI work. The team has continued to build on that foundation as its training workloads have grown.

With Anyscale, Chef Robotics is able to:

• Move from multi day training cycles toward same day iteration. Chef trains and fine tunes foundation models like Pi0 on Anyscale, launching jobs through the Anyscale CLI and monitoring progress through Weights and Biases and Anyscale's own tooling, resulting in an order of magnitude reduction in total training time.

• Scale distributed training reliably from single node to multi node GPU clusters. Chef now trains its foundation models on 16 A100 GPU clusters on Anyscale, with the platform managing GPU access across regions automatically rather than leaving the team to manually search for available capacity.

• Move between spot and reserved instances without absorbing the operational cost of interruptions. Chef can shift training jobs between spot and reserved capacity with a configuration change rather than a manual process, letting the team take advantage of spot pricing when it makes sense without engineering around preemptions themselves.

"We can throw a large machine learned model at the problem and deal with corner cases that would be impossible to reason about with a classical, physics based model. That takes many parameters and a lot of data, and you cannot take it on with smaller approaches."

Luis Rayas | CTO, Chef Robotics

LinkFaster iteration and developer experience 

Before Anyscale, testing a new foundation model meant manually provisioning GCP instances and kicking off training scripts by hand, a workflow that held up well enough for early experimentation but added friction as the pace of iteration picked up. On Anyscale, Chef's team launches every training job through the Anyscale CLI, giving the team a consistent, scriptable way to start runs against its GPU clusters instead of managing instances directly. Because the CLI and the underlying Anyscale API are straightforward to call programmatically, the team has been able to build its own tooling around that workflow, launching jobs and checking status without touching infrastructure by hand.

That same visibility carries into how the team handles failures. When a training job fails partway through, typically after running for about an hour, the cause has so far always come down to a memory related issue, though the specific fix has differed each time. Anyscale's dashboards, alongside Weights and Biases, give the team a consistent view into cluster, resource, and job level detail, which has made it straightforward to track a failure down and get back to a working run without extended downtime.

"When a training job failed, I would have an agent help debug it directly through the Anyscale CLI. Both times the failures were memory related, but the fixes were different each time, and having an agent in the driver's seat made it easy to work through."

Luis Rayas | CTO, Chef Robotics

LinkReliable GPU access from single node to distributed scale

GPU availability was one of the more unpredictable parts of Chef's early training work. Losing access to a cluster meant a real risk of waiting days to regain H100 capacity, which is why the team kept its workspace active around the clock rather than risk that delay. As training has scaled onto 16 A100 GPU clusters, Anyscale has taken over much of that burden directly, searching across regions for available capacity rather than leaving the team to do that manually.

Throughout that growth, GPU utilization has stayed high. The team tracks utilization directly in Anyscale and has kept GPUs fully saturated on a single machine both before and after adopting the platform, which means scaling up to larger clusters has not come at the cost of efficiency.

"Early on, losing our GPU cluster meant a real risk of not getting H100 access back for days. Now with Anyscale, if a region does not have capacity, we can try others automatically until one comes through. Additionally, we track GPU utilization directly in Anyscale, which stays fully saturated the entire time at greater scale."

Luis Rayas | CTO, Chef Robotics

LinkFlexible spot and reserved capacity without the preemption tax

Chef currently runs its foundation training primarily on reserved instances, largely because data collection is ad hoc and the team needs to start training as soon as new data is ready. Earlier in Chef's use of Anyscale, the team also tested spot instances directly, and unlike its experience running spot capacity on raw GCP, where preemptions could cause job crashes and force manual restarts, spot instances on Anyscale stayed reliable, giving the team room to shift toward spot pricing for cost efficiency without taking on the engineering work of managing interruptions themselves.

"Switching between spot and on demand instances both worked well on Anyscale. In the past it was a hassle to get instances reserved, but here I can flip a setting and let Anyscale handle it."

Luis Rayas | CTO, Chef Robotics

LinkWhat's Next

Chef is preparing to expand its use of Anyscale into a new phase of research, training a predictive world model for its robots. Parallelizing simulation workloads on Anyscale is on Chef's roadmap as well, and the team expects its platform organization to take on shared ownership of Anyscale over time as it moves from an applied AI research tool toward part of Chef's core production infrastructure.

"Our next big experiment is training a predictive world model, essentially a neural simulator that lets us run counterfactuals and train our systems to perform tasks better before they ever touch a robot."

Luis Rayas | CTO, Chef Robotics