Question 1

What's the difference between model inference and model training?

Accepted Answer

Training is when you teach a model by learning patterns from historical data (it’s compute-heavy and can take minutes to days). Inference is when you use the already-trained model to make predictions on new data (often needs low latency, like milliseconds to seconds, or runs in batch).

Question 2

When should I use real-time (online) inference vs batch inference?

Accepted Answer

Use real-time inference when you need an immediate response per request (chatbots, fraud checks at checkout, image classification in an app). Use batch inference when you can process many records at once on a schedule (nightly churn scoring, weekly demand forecasts, backfilling predictions for analytics).

Question 3

How much does model inference cost?

Accepted Answer

Cost depends on (1) compute type (CPU vs GPU/accelerators), (2) how long endpoints run (always-on vs scale-to-zero options), (3) traffic volume and peak concurrency, (4) model size and latency requirements, (5) batch job duration, and (6) data transfer and storage. Real-time endpoints typically charge per instance-hour (plus any accelerator costs), while batch inference charges for the compute time used during the job.

Model Inference

Definition

Real-World Example

Related Terms

Cloud Provider Equivalencies

Explore More Cloud Computing Terms