NeMo Microservices — NVIDIA NeMo Microservices Documentation
Introduction
NeMo Microservices give you the infrastructure to build and deploy specialized AI agents with open source models. They provide
,
and
,
, real-time protection with
, and
. Production-grade features include
and
.
or
,
integrate with your existing tools
, and customize models for your specific use cases while maintaining control over your AI stack.
Common use cases
Customize and evaluate models — Generate synthetic training data, fine-tune models, and measure quality. See
for workflows like creating text-to-code datasets and fine-tuning with synthetic data.
Deploy and serve models — Run inference through the unified gateway and integrate with your existing infrastructure. See
and the
for deployment examples.
Test and protect AI agents — Scan for vulnerabilities with
, then block attacks in real-time with
.
Build RAG and search applications — Fine-tune
for domain-specific retrieval and evaluate with
.
Getting up and running
Prerequisites:
Python 3.11+ and pip (or
)
28.3.0+
The
and an NGC API key with access to the early access org for NeMo Microservices (0857255566152269)
A
API token (used for cloud inference, separate from the NGC key)
Hardware and Software Requirements for NeMo Microservices
Download the SDK from the NGC private registry and install it:
exportNGC_CLI_API_KEY=<your-ngc-api-key> ngcregistryresourcedownload-version"0857255566152269/external/nemo-platform-python-sdk:2.0.1" pipinstallnemo-platform-python-sdk_v2.0.1/*.whl Pull the platform image, then start the
(local platform):
echo"${NGC_CLI_API_KEY}"|dockerloginnvcr.io-u'$oauthtoken'--password-stdin dockerpullnvcr.io/0857255566152269/external/nmp-api:26.03.1 nmpquickstartconfigure--auto nmpquickstartup--imagenvcr.io/0857255566152269/external/nmp-api:26.03.1 For full setup — including the task images that services launch on demand — see
. Once the platform is running:
List available models and other key commands:
nmpmodelslist# See available models nmpchat--help# Chat options nmpworkspaceslist# View your workspaces nmp--help# All commandsFor full installation steps, GPU config, and SDK usage, see
; for all commands, see
.
Before you start
Workspaces — All platform resources (models, datasets, jobs, evaluation results) belong to a
. Workspaces provide organizational and authorization boundaries—create separate workspaces to isolate teams, users, environments, or clients. The platform includes two built-in workspaces: default (general-purpose, editable by all) and system (read-only platform resources). When
is enabled, users are granted roles (Viewer, Editor, or Admin) within specific workspaces. See
for creating and managing workspaces.
Projects — Group related resources with
. Projects are organizational tags within a workspace, useful for fine-tuning experiments, evaluation campaigns, or other work within a team. Access control applies at the workspace level, not per project.
Entities — Models, datasets, jobs, and configurations are
—the shared data objects that power platform services. See the entities page for how they’re stored, scoped, and used.
Where to go next
Start building:
— End-to-end workflows combining multiple platform capabilities
— Quickstart Installation (GPU configuration and SDK setup)
Learn the platform:
— Workspaces, projects, and entity organization
— CLI reference and configuration
NeMo Microservices API Reference
— REST API reference
Deploy to production:
— Deploy on Kubernetes with Helm
Authentication and Authorization
— Configure role-based access control
Architecture
Figure 1 Click to view full size.