Dockers_dev/ollama/rag/md_only/DockerSwarm.md

Docker Swarm

This installation method requires knowledge on Docker Swarms, as it utilizes a stack file to deploy 3 seperate containers as services in a Docker Swarm.

It includes isolated containers of ChromaDB, Ollama, and OpenWebUI. Additionally, there are pre-filled Environment Variables to further illustrate the setup.

Choose the appropriate command based on your hardware setup:

• Before Starting:

Directories for your volumes need to be created on the host, or you can specify a custom location or volume.

The current example utilizes an isolated dir data, which is within the same dir as the docker-stack.yaml.

  - **For example**:

    ```bash
    mkdir -p data/open-webui data/chromadb data/ollama
    ```

• With GPU Support:

#### Docker-stack.yaml

```yaml
version: '3.9'

services:
  openWebUI:
    image: ghcr.io/open-webui/open-webui:main
    depends_on:
        - chromadb
        - ollama
    volumes:
      - ./data/open-webui:/app/backend/data
    environment:
      DATA_DIR: /app/backend/data 
      OLLAMA_BASE_URLS: http://ollama:11434
      CHROMA_HTTP_PORT: 8000
      CHROMA_HTTP_HOST: chromadb
      CHROMA_TENANT: default_tenant
      VECTOR_DB: chroma
      WEBUI_NAME: Awesome ChatBot
      CORS_ALLOW_ORIGIN: "*" # This is the current Default, will need to change before going live
      RAG_EMBEDDING_ENGINE: ollama
      RAG_EMBEDDING_MODEL: nomic-embed-text-v1.5
      RAG_EMBEDDING_MODEL_TRUST_REMOTE_CODE: "True"
    ports:
      - target: 8080
        published: 8080
        mode: overlay
    deploy:
      replicas: 1
      restart_policy:
        condition: any
        delay: 5s
        max_attempts: 3

  chromadb:
    hostname: chromadb
    image: chromadb/chroma:0.5.15
    volumes:
      - ./data/chromadb:/chroma/chroma
    environment:
      - IS_PERSISTENT=TRUE
      - ALLOW_RESET=TRUE
      - PERSIST_DIRECTORY=/chroma/chroma
    ports: 
      - target: 8000
        published: 8000
        mode: overlay
    deploy:
      replicas: 1
      restart_policy:
        condition: any
        delay: 5s
        max_attempts: 3
    healthcheck: 
      test: ["CMD-SHELL", "curl localhost:8000/api/v1/heartbeat || exit 1"]
      interval: 10s
      retries: 2
      start_period: 5s
      timeout: 10s

  ollama:
    image: ollama/ollama:latest
    hostname: ollama
    ports:
      - target: 11434
        published: 11434
        mode: overlay
    deploy:
      resources:
        reservations:
          generic_resources:
            - discrete_resource_spec:
                kind: "NVIDIA-GPU"
                value: 0
      replicas: 1
      restart_policy:
        condition: any
        delay: 5s
        max_attempts: 3
    volumes:
      - ./data/ollama:/root/.ollama

```

• Additional Requirements:

  1. Ensure CUDA is Enabled, follow your OS and GPU instructions for that.
  2. Enable Docker GPU support, see Nvidia Container Toolkit
  3. Follow the Guide here on configuring Docker Swarm to with with your GPU
  4. Ensure GPU Resource is enabled in /etc/nvidia-container-runtime/config.toml and enable GPU resource advertising by uncommenting the swarm-resource = "DOCKER_RESOURCE_GPU". The docker daemon must be restarted after updating these files on each node.

• With CPU Support:

  Modify the Ollama Service within docker-stack.yaml and remove the lines for generic_resources:

      ollama:
    image: ollama/ollama:latest
    hostname: ollama
    ports:
      - target: 11434
        published: 11434
        mode: overlay
    deploy:
      replicas: 1
      restart_policy:
        condition: any
        delay: 5s
        max_attempts: 3
    volumes:
      - ./data/ollama:/root/.ollama
  

• Deploy Docker Stack:

  docker stack deploy -c docker-stack.yaml -d super-awesome-ai