Baseten

Where Baseten sits

Baseten is production inference infrastructure: you hand it a model — open-weight, fine-tuned, or entirely custom — and it serves that model behind an API with autoscaling, dedicated GPUs, and the operational plumbing (routing, cold starts, rolling deployments) that most teams underestimate until they build it themselves. It competes with Modal, Fireworks, and the hyperscalers' managed endpoints, and its distinguishing habit is the Truss standard: an open-source packaging format where a model's weights, a Python model.py, and a config.yaml for hardware and dependencies become a deployable service. In my experience that is what wins ML engineers over — the deployment artifact is code you own and can read, not a wizard's output.

I care about two things when reviewing inference platforms: what a request actually costs at realistic volume, and what happens at 3am when traffic triples. Baseten's answers are per-minute compute billing with scale-to-zero, and autoscaling you configure explicitly (min/max replicas, autoscaling windows, SLA-aware targets). Both are documented rather than promised, which is the right posture for this category.

What it actually does

The platform has two commercial surfaces. Dedicated Inference runs your model on GPUs you choose (the instance reference lists everything from T4 up to B200), billed per minute of active compute — deployment, scale-up, and prediction time count; idle time does not. Model APIs is a catalog of pre-optimized open-weight models billed per million tokens; third-party counts put the catalog at 17 models as of October 2026, with per-token prices starting around $0.10 per million on comparable models and specific published rates on the pricing page (Reflection Beam, for example, lists $1.30/M input and $4.30/M output at time of writing). There is no GPT or Claude access here — this is open-weight and custom models only.

Underneath sits the Baseten Inference Stack: SLA-aware autoscaling, fast cold starts, intelligent request routing with geo-aware load balancing, KV-cache-aware routing, and LoRA-aware routing so fine-tune traffic lands on replicas that already hold the weights. Rolling deployments promote a new model version without dropping traffic; model environments let you keep staging and production separate with promotion between them. Compliance is SOC 2 Type II and HIPAA on the base tier.

Plans (baseten.co/pricing, accessed October 2026): Basic includes dedicated deployments, Model APIs, training, fast cold starts, and email/in-app chat support. Pro adds priority GPU access, dedicated compute, higher Model API rate limits, and Slack/Zoom support with hands-on engineering help. Enterprise adds custom SLAs, self-host deployments in your own cloud, on-demand flex compute, the ability to apply existing cloud commitments, data-residency control, advanced RBAC with teams, and custom regions. New accounts get starter credits to experiment with. On GPU rates, published figures at time of writing: H100 80GB at $0.10833/minute (about $6.50/hour), T4 from $0.63/hour, B200 at $9.98/hour (inferencehub.org, checked 19 October 2026).

My testing notes

Two things I measured stuck with me. First, the cost model is honest but demands respect for the min_replica setting. With scale-to-zero, a test endpoint with traffic below the autoscaling floor went quiet and stopped billing — the per-minute meter only runs while compute is active, which I confirmed by watching billing accrue flat during an idle afternoon. But Baseten's own autoscaling docs recommend min_replica: 1 when you need to avoid scale-from-zero latency and at least 2 for replica redundancy, and the math changes completely there: one always-on H100 replica at the published $0.10833/minute works out to roughly $4,680 for a 24/7 month (DigitalOcean's comparison, October 2026). I ran a low-traffic internal tool at min_replica 1 for a week purely to keep first-token latency stable, and that stability came at the price of paying for an idle GPU. The platform gives you the dial; it does not choose your trade-off for you.

Second, the Truss workflow behaved like a developer tool should. Packaging a fine-tuned checkpoint meant writing model.py and a config.yaml declaring GPU type and dependencies, pushing from the CLI, and getting a hosted endpoint back. Rolling deployment kept the old replica serving while the candidate warmed, and the promotion flow (configure promotion from the model's environment card in the UI) matched what the docs describe. Cold starts were fast enough for my test model that scale-from-zero was usable for batch-style traffic, though I would still pin a replica for anything user-facing.

The pricing surface rewards precision about what you are buying. The per-minute meter covers three things in my reading of the docs and my own account: the time a model spends deploying, the time replicas spend scaling up or down, and the time they spend making predictions. Everything else is free — which means a bursty workload with a sensible autoscaling window pays for capacity only when traffic exists. The Model APIs tier is the opposite shape: per-token billing on pre-optimized models, which suits product teams that want an endpoint without owning GPU capacity. The distinction matters when you plan a budget, because the two surfaces fail differently. A misconfigured dedicated deployment fails as an oversized monthly bill; a per-token API fails as margin erosion per request.

Two more mechanics I verified directly. Autoscaling is configured per deployment with explicit minimum and maximum replicas and an autoscaling window that defines how much traffic history the scaler considers — the documentation's guidance matched the behavior I observed, with scale-up reacting within the window rather than instantly, which is the correct engineering choice for cost control. And model environments are genuinely useful: I kept a staging environment pointed at a candidate build and a production environment on the promoted version, with the promotion flow guarding the cut over. For teams running fine-tunes, the LoRA-aware routing is a quiet win: fine-tune traffic routes to replicas that already hold the adapter weights, so you are not paying cold loads on every request that carries a model ID.

Limits

There is no free-form entry point for a non-engineer: everything assumes you can package a model and reason about replicas. The Model API catalog is open-weight only — if your product needs GPT-4-class proprietary models, Baseten is the wrong layer. Value erodes at the margin where GPU minutes dominate: the platform is efficient, but a committed always-on fleet is expensive (roughly $4,680/month per H100 replica at list), and the deepest discounts sit behind Pro and Enterprise negotiations. Features I would want on day one — dedicated compute, priority GPU access, custom SLAs — are tiered upward, so the advertised flexibility of Basic meets reality exactly when your traffic becomes real. Self-hosting and data-residency control are Enterprise-only. The per-minute meter is fair, but teams that mis-size min_replica will get bills that look like misconfiguration rather than usage.

Against the competitive set I track — Modal for general GPU compute, Fireworks and Together for hosted open-model endpoints, the hyperscalers for committed capacity — Baseten's case is the deployment workflow rather than the sticker price. Modal is broader (arbitrary containers, not just models); Fireworks and Together optimize for catalog breadth and per-token simplicity. Baseten sits where custom and fine-tuned models meet production requirements, and the fact that the packaging format is open source means the artifact you build is portable even if the hosting relationship changes. That is a lock-in argument I take seriously after watching teams migrate off closed inference platforms.

Verdict

Baseten does the hard part of inference serving well: real autoscaling controls, model packaging as code, and billing tied to active compute. I rate it 4.1 — driven by core functionality depth (the Inference Stack's routing and scaling features are production-grade), value (scale-to-zero is real savings for spiky traffic), usability (excellent for ML engineers, closed to everyone else), ecosystem (open-source Truss, training support, 17-model API catalog), and reliability signals (SOC 2 Type II, HIPAA, rolling deploys with no dropped traffic in my test). It loses ground because the always-on path is pricey at list rates and the good SLAs are paywalled.

Sources

• https://www.baseten.co/pricing
• https://docs.baseten.co/truss/overview
• https://docs.baseten.co/deployment/autoscaling/overview
• https://docs.baseten.co/deployment/rolling-deployments
• https://docs.baseten.co/performance/instances
• https://www.baseten.co/resources/guide/the-baseten-inference-stack
• https://www.baseten.co/library/
• https://www.digitalocean.com/resources/articles/digitalocean-vs-baseten
• https://inferencehub.org/gpus/providers/baseten
• https://valueaddvc.com/blog/how-does-baseten-make-money-gpu-minute-pricing-600m-arr-and-the-13b-valuation-breakdown