Guides on Modelplane Docs

Anthropic Messages API

An InferenceGateway serves a ModelService whose backends speak OpenAI on both APIs, translating an Anthropic request to OpenAI, so one model answers OpenAI at /v1/chat/completions and Anthropic at /anthropic/v1/messages. A client that speaks the Messages API, including Claude Code via ANTHROPIC_BASE_URL, names the service as the model in its request. See Alternate APIs for the detail.

This recipe serves Qwen3-8B on one NVIDIA H100 on Nebius, with tool calling on: --enable-auto-tool-choice and --tool-call-parser=hermes are what let Claude Code’s tool use work. An 8B model needs a fraction of an H100, so the GPU has ample headroom. Apply the platform side first, then the ML side.

Multi-node serving on Dynamo

Qwen2.5-14B’s FP16 weights are about 29 GB, larger than one NVIDIA L4’s 23 GB, so it serves across two nodes as a gang: a Leader and a Worker, one L4 each, pipeline-parallel across the pair. On a Dynamo cluster Grove and the KAI Scheduler gang-schedule the two pods together, Modelplane composes them as a Grove PodCliqueSet, and NVIDIA ModelExpress serves their weights.

This is the getting started tour scaled to two nodes: one larger model on a spec.stack: Dynamo cluster. Set up the platform first, for the gateway and cloud credentials, then apply the manifests below.

An Open Field Guide to Inference Engineering

Note

Numbers, prices and tool versions are as of October 2026.

This guide is for platform engineers new to inference. If you need to understand how models and GPUs work together and how to run inference in production, this guide will help. You’ll also learn about the tools to use and how to grow into an inference engineering role.

This guide builds on the work of others in the inference community. If you’re interested in inference engineering, continue your learning with these: