GPU workers that spin up when a request arrives and disappear when it's done. No idle instances, no capacity planning, no invoice for a machine that sat there overnight.
Serverless sits on the same 20,000+ GPU marketplace as everything else we run — from consumer cards to B200 nodes. You describe the workload; we pick the hardware, scale it, and bill by the second.
Define the image and the scaling rules once. Worker lifecycle, placement and replacement are handled for you.
The same marketplace price you'd pay renting the card directly. No serverless surcharge, no pro tier, no per-request fee.
RTX 4090 for cheap throughput, H200 or B200 when the model demands it. Pin a GPU type or let us match one to the job.
Place workers near your users to cut latency, or inside a region because the data isn't allowed to leave it.
Inference traffic doesn't arrive evenly. Workers follow the curve instead of sitting at peak capacity all day.
Scale on queue depth, request rate or latency target — whichever actually predicts your load. Down to zero when the queue empties.
Keep a few workers pre-loaded so the first request after a quiet period doesn't wait on a container pull. Set the floor to zero if cost matters more than the first-token time.
vLLM, TGI, ComfyUI, or your own image from a private registry. If it runs in Docker and listens on a port, it runs here.
Per-worker logs and utilisation, plus SSH into a live worker when something only reproduces in production.
Image, scaling policy and worker types live in your repo, not in a dashboard someone clicked through once and forgot.
Route cheap requests to 4090s and heavy ones to H200s behind a single endpoint, with its own scaling rule per type.
The endpoint, the image, the scaling policy and the worker mix are all one object in your codebase. Review it, diff it, roll it back — same as the rest of your infrastructure.
from openlink import OpenLink
ol = OpenLink(api_key="...")
ep = ol.serverless.create_endpoint(
name="embeddings",
image="vllm/vllm-openai:latest",
port=8000,
workers=[
{"gpu": "RTX_4090", "min": 0, "max": 40},
{"gpu": "H200", "min": 1, "max": 8},
],
scale_on={"queue_depth": 4},
idle_timeout=120, # 秒 — 空闲后回收
)
print(ep["url"])
# https://embeddings-a41f.openlink.ai/v1The pattern is common enough to be worth naming: managed GPU platforms tend to charge a premium for the convenience, then limit which hardware you can use.
| Feature | OpenLink Serverless | Common elsewhere |
|---|---|---|
| Pricing | Marketplace rate, no serverless surcharge | Premium tier on top of the hourly rate |
| Scale to zero | Yes — idle timeout you set | Often a paid minimum floor |
| Hardware choice | The full fleet, consumer through B200 | A short list of approved SKUs |
| Cold starts | Configurable warm pool | Cold on every quiet period |
| Worker types | Several GPU types per endpoint | One type per endpoint |
| Debugging | Logs, metrics and SSH into a live worker | Logs only, if that |
| Configuration | Defined in code, lives in your repo | Dashboard-first, hard to review |
| Billing granularity | Per second | Per minute or per hour |
Right-hand column describes patterns we see across managed GPU platforms generally, not any one named provider. Terms change — check the current pricing page of anything you're comparing against.
Serverless doesn't mean shared. Every worker is an isolated instance on dedicated hardware, torn down when it scales in.
Dedicated hardware per worker with direct SSH. No container sharing between tenants.
Worker storage is destroyed on scale-in. Anything you need to keep goes to a volume you control.
Constrain an endpoint to a region or a named facility when residency rules dictate where the data can sit.
Separate keys per endpoint or environment, each with its own spend cap. Revoke one without touching the rest.
min above zero on a worker group keeps that many workers warm so the first request after a quiet period doesn't pay the pull. Setting it to zero costs nothing when idle but makes the first request slower.ol.serverless.create_endpoint() from the Python SDK, or the equivalent from the CLI. The dashboard exists for inspecting what's running, not as the only way to change it.One OpenAI-compatible endpoint for 300+ models, with routing and failover.
API Gateway →Search GPUs, deploy containers and script the whole workflow from your shell.
OpenLink CLI →A typed client for compute and models. Build agents that scale their own GPUs.
Python SDK →Skip the quotas, skip the contracts, skip the chaos. Scale when the traffic does, and stop paying when it stops.