LitServe

LitServe is a Python framework that enables teams to build and run custom AI inference servers and APIs, providing components for request handling, batching, streaming, and scaling while supporting self-hosted and vendor-hosted deployment models.