Inference
-
Inference Cost Optimization: Autoscaling, Batching, Spot
Inference cost is dominated by idle capacity and underused accelerators, not per-request price. Autoscaling signals, dynamic batching and spot capacity.
-
Model Serving Compared: SageMaker, Vertex AI, Databricks
All three managed platforms serve a model behind an endpoint. The differences that matter show up in autoscaling, multi-model density, and data coupling.
-
Online Inference Latency: Where the Budget Actually Goes
P99 latency is a product problem as much as an engineering one. Breaking down the inference budget: model compute, preprocessing, retrieval and network.