Model serving is the process of deploying a trained AI model behind an API endpoint so that applications can send requests and receive predictions in real time. It involves batching incoming requests for GPU efficiency, managing multiple model versions, scaling based on traffic, and monitoring latency. Platforms like NVIDIA Triton, TensorFlow Serving, and tools like BentoML and vLLM provide production-grade serving infrastructure.
AI Infrastructure
Model Serving
Related terms
Learn to use these concepts in practice.
Join the 212AY Academy →