Docker Setup

Kuzco Docker

Welcome to the Kuzco Docker setup guide! This document provides detailed instructions for deploying Kuzco Workers using Docker on both Windows and Linux systems, supporting both NVIDIA and AMD GPUs. Follow these steps to install Docker, configure the necessary GPU runtimes, and launch your Kuzco Worker instances efficiently. Whether you're utilizing multiple GPUs or specifying individual ones, this guide will help you maximize your hardware capabilities. Additionally, we've included useful commands and extra helper scripts to streamline your setup and maintenance process. For visual learners, reference images and an Asciinema recording are provided. Join our Discord community if you need further assistance.

Getting Started

  1. Download and install Docker or Docker Desktop
  2. Navigate and login to https://kuzco.xyz (opens in a new tab)
  3. Select Workers, then continue to "Create Worker"
  4. Select Docker Worker
  5. After, click "Launch Worker" and you will see instructions and commands to use with Docker for either NVIDIA or AMD Kuzco Docker Containers

image image

Windows (NVIDIA)

  1. Download and Install Docker

  2. Install NVIDIA Container Toolkit

  3. Pull Docker Image

    docker pull kuzcoxyz/worker:latest

    Docker image download may take a few minutes to complete. (Model Weights included)

  4. Run NVIDIA Worker Instance

(You can run this command as many times as your GPU or GPUs will fit) (e.g. Llama3 8B takes roughly ~5GB of VRAM, so if you have a 12GB VRAM GPU, you can run this following command two times to run two instances)

docker run --rm --runtime=nvidia --gpus all -d kuzcoxyz/worker:latest --worker <workerid> --code <code>

To specify a particular GPU, use (0 being GPU #0):

docker run --rm --runtime=nvidia --gpus '"device=0"' -d kuzcoxyz/worker:latest --worker <workerid> --code <code>
For example, '"device=2,3"' will enumerate GPUs 2 and 3 to the container.

asciicast (opens in a new tab)

Once your docker instances are running you will see them in your Worker on https://kuzco.xyz (opens in a new tab)

image

Linux (NVIDIA and AMD)

  1. Download and Install Docker

  2. For NVIDIA: Install NVIDIA Container Toolkit

  3. Pull Docker Image

    docker pull kuzcoxyz/worker:latest

    Docker image download may take a few minutes to complete. (Model Weights included)

  4. Run NVIDIA Worker Instance

(You can run this command as many times as your GPU or GPUs will fit) (e.g. Llama3 8B takes roughly ~5GB of VRAM, so if you have a 12GB VRAM GPU, you can run this following command two times to run two instances)

docker run --rm --runtime=nvidia --gpus all -d kuzcoxyz/worker:latest --worker <workerid> --code <code>

To specify a particular GPU, use (0 being GPU #0):

docker run --rm --runtime=nvidia --gpus '"device=0"' -d kuzcoxyz/worker:latest --worker <workerid> --code <code>
For example, '"device=2,3"' will enumerate GPUs 2 and 3 to the container.

asciicast (opens in a new tab)

Once your docker instances are running you will see them in your Worker on https://kuzco.xyz (opens in a new tab)

image

  1. For AMD: Install ROCm and Configure Docker

  2. Run AMD Worker Instance

(You can run this command as many times as your GPU or GPUs will fit) (e.g. Llama3 8B takes roughly ~5GB of VRAM, so if you have a 12GB VRAM GPU, you can run this following command two times to run two instances)

docker run --rm --device=/dev/kfd --device=/dev/dri --group-add video --group-add 110 --security-opt seccomp=unconfined -d kuzcoxyz/workeramd:latest --worker <workerid> --code <code>

Useful Commands

To list all running Docker containers:

docker ps

To stop all running Docker containers:

docker stop $(docker ps -q)

Extra Helper Scripts

Run the below script with a crontab job to check for Docker image updates automatically (for NVIDIA only).

#!/bin/bash
 
# Variables
IMAGE_NAME="kuzcoxyz/worker:latest"
NUM_CONTAINERS=${1:-1}
WORKER=${2:-"default_worker"}
CODE=${3:-"default_code"}
 
# Command arguments
COMMAND_ARGS="--worker $WORKER --code $CODE"
 
# Pull the latest image
docker pull $IMAGE_NAME
 
# Function to start a container with a specific GPU
start_container() {
    local container_name=$1
    local gpu_device=$2
    docker run --rm --runtime=nvidia --gpus "device=$gpu_device" -d --name $container_name $IMAGE_NAME $COMMAND_ARGS
}
 
# Stop and remove all running containers of the specified image
running_containers=$(docker ps --filter "ancestor=$IMAGE_NAME" -q)
if [ ! -z "$running_containers" ]; then
    docker stop $running_containers
fi
 
sleep 5
 
# Get the list of available GPU devices
gpu_devices=$(nvidia-smi --query-gpu=index --format=csv,noheader)
gpu_array=($gpu_devices)
num_gpus=${#gpu_array[@]}
 
# Start the specified number of containers, evenly distributing GPUs
for i in $(seq 1 $NUM_CONTAINERS); do
    gpu_index=$(( (i - 1) % num_gpus ))
    start_container "${IMAGE_NAME//[:\/]/_}_$i" "${gpu_array[$gpu_index]}"
done
 
echo "$NUM_CONTAINERS containers started with image $IMAGE_NAME using worker $WORKER and code $CODE."

If you have any issues, join our Discord for help.

WSL SetupKuzco CLI