The ACE Controller SDK allows you to build your own ACE Controller service to manage multimodal, real-time interactions with voice bots and avatars using NVIDIA ACE. With the SDK, you can create controllers that leverage the Python-based open-source Pipecat framework for creating real-time, voice-enabled, and multimodal conversational AI agents. The SDK contains enhancements to the Pipecat framework, enabling developers to effortlessly customize, debug, and deploy complex pipelines while integrating robust NVIDIA Services into the Pipecat ecosystem.
-
Pipecat Extension: A Pipecat extension to connect with ACE services and NVIDIA NIMs, facilitating the creation of human-avatar interactions. The NVIDIA Pipecat library augments the Pipecat framework by adding additional frame processors and services, as well as new multimodal frames to enhance avatar interactions. This includes the integration of NVIDIA services and NIMs such as NVIDIA Riva, NVIDIA Audio2Face, and NVIDIA Foundational RAG.
-
HTTP and WebSocket Server Implementation: The SDK provides a FastAPI-based HTTP and WebSocket server implementation compatible with ACE. It includes functionality for stream and pipeline management by offering new Pipecat pipeline runners and transports. For ease of use and distribution, this functionality is currently included in the
nvidia-pipecatPython library as well.
The ACE Controller SDK was used to build the ACE Controller Microservice.Check out the ACE documentation for more details on how to configure the ACE Controller MS with your custom pipelines.
The NVIDIA Pipecat package is released as a wheel on PyPI. Create a Python virtual environment and use the pip command to install the nvidia-pipecat package.
pip install nvidia-pipecatYou can start building pipecat pipelines utilizing services from the NVIDIA Pipecat package. For more details, follow the ACE Controller and the Pipecat Framework documentation.
If you wish to work directly with the source code or modify services from the nvidia-pipecat package, you can utilize either the UV or Nix development setup as outlined below.
To get started, first install the UV package manager.
Then, create a virtual environment with all the required dependencies by running the following commands:
uv venv
uv sync
source .venv/bin/activateOnce the environment is set up, you can begin building pipelines or modifying the services in the source code.
If you wish to contribute your changes to the repository, please ensure you run the unit tests, linter, and formatting tool.
To run unit tests, use:
uv run pytest
To format the code, use:
ruff formatTo run the linter, use:
ruff check
To set up your development environment using the Nix, follow these steps:
Initialize the development environment: Simply run the following command:
nix developThis setup provides you with a fully configured environment, allowing you to focus on development without worrying about dependency management.
To ensure that all checks such as the formatting and linter for the repository are passing, use the following command:
nix flake checkWe invite contributions! Open a GitHub issue or pull request! See contributing guildelines here.