There are four common patterns of machine learning production: pipeline, ensemble, business logic, and online learning. Implementing these patterns typically involves a tradeoff between ease of development and production readiness.
Web frameworks are simple and work out of the box but can only provide single predictions; they cannot deliver performance or scale. Custom tooling glue tools together but are hard to develop, deploy, and manage. Specialized systems are great at serving ML models but they are not as flexible or easy to use and can be costly.
Anyscale helps you go beyond existing model serving limitations with Ray and Ray Serve, which offers scalable, efficient, composable, and flexible serving. Ray Serve provides:
Effective machine learning serving frameworks need to be open to meet different demands. Ray Serve allows you to bring your own Docker, is multi-framework (e.g., TF, PyTorch, Sklear, XGboox, etc.), offers different runtime environments per tasks and actors, and different framework versions running on each task and actor.
With native support for FastAPI, Ray Serve allows you to bridge the gap between web server and specialized model serving frameworks. Leverage automatic documentation, typed python (Pydantic), validation, security and authentication, performance, asynchronicity, and routing.
Chaining, parallelization, ensemble, and dynamic dispatch patterns can be easily expressed with plain Python code. Test locally and deploy to production with no cde changes and different runtime environments per tasks and actors. Clearly define the separation and boundaries between code and deployments.
By building on top of Ray, Ray Serve is horizontally scalable, lightning fast, and efficient by allowing fractional and fine-grained resource allocation.
Serve machine learning models in real-time or batch using a simple Python API
Already using open source Ray?
Migrate your existing workloads to Anyscale with no code changes. Experience the magic of infinite scale at your fingertips.