Route model requests with NeMo Switchyard

Follow a request from a video analysis app to a model endpoint and see what to check before serving it.

NeMo Switchyard project artwork
Official visualNVIDIA NeMo

Model routing for LLM agents and applications.

Switchyard components

The official diagram separates the host or gateway from model selection and upstream calls. Official figure source

Architecture and data flow

How the data moves
  1. InputPrepare an app requestobservation → summary request

    The app turns a video observation into a text task.

  2. ProcessSelect a modelrequest + model pool + routing strategy

    Switchyard helps the host select the model for the request.

  3. OutputCall the endpointmodel selection → response

    The host calls the selected provider and the app uses its response.

Switchyard separates model choice from the application’s task logic. A host gives the routing library a request, a pool of available models and a strategy. Task routing can ask a model to classify the request, and stage routing can take recent tool activity into account. The host handles requested upstream calls and consumes the routing outcome. Gateway integrations can translate OpenAI and Anthropic request, response and streaming formats.

Where does a request go after video analysis?

Suppose a video service detects a maintenance event at 00:12. The app turns that observation into a text summary request. The host uses Switchyard with the request, available models, and routing strategy to choose a model. The app uses the selected endpoint's text to finish the report. Record the clip ID, selected model, response time, and report completion together to see which stage needs attention. Official Switchyard repository

Physical AI example

For an inspection assistant, keep image and video analysis in a vision-capable service such as VSS. Route its downstream text summarization and report drafting across a model pool. Compare a fixed single-model baseline with the routed setup on the same clips and prompts.

What to verify

Measure cost and latency per successfully completed report. Verify tool calls, streaming, provider error handling and accepted modalities end to end. The standalone server is documented for demos and local evaluation. Check the gateway integration and its version before deployment.

License and availability

Apache-2.0 repository code. Model weights, external services and dependencies can have separate terms.

Official sources

NeMo Relay · Metropolis / VSS