hops/ADVANCED.md

Advanced Configuration & Troubleshooting

This guide covers advanced HOPS configuration, troubleshooting, security features, and system administration.

🔧 Advanced Configuration

Environment Variables

HOPS uses both encrypted secret management and traditional environment files for configuration.

Encrypted Secret Management (v3.1.0+)

# Initialize secret management
./lib/secrets.sh init

# Create encrypted environment
./lib/secrets.sh create

# Update values
./lib/secrets.sh update DOMAIN example.com
./lib/secrets.sh update ACME_EMAIL admin@example.com

# Retrieve values
./lib/secrets.sh get PUID
./lib/secrets.sh get DEFAULT_ADMIN_PASSWORD

# List all keys
./lib/secrets.sh list

# Backup encrypted secrets
./lib/secrets.sh backup /backup/location/

Traditional Environment File (~/hops/.env)

# User and Group Configuration
PUID=1000                     # User ID
PGID=1000                     # Group ID
TZ=America/New_York          # Timezone

# Directory Configuration
DATA_ROOT=/mnt/media         # Media storage (Linux)
CONFIG_ROOT=/opt/appdata     # App configurations (Linux)

# macOS paths (automatically set)
DATA_ROOT=/Users/username/hops/media
CONFIG_ROOT=/Users/username/hops/config

# Security (auto-generated and encrypted)
DEFAULT_ADMIN_PASSWORD=...   # Generated secure password
DEFAULT_DB_PASSWORD=...      # Database password
JELLYFIN_PASSWORD=...        # Service-specific passwords

# Optional: Custom Domain Configuration
DOMAIN=yourdomain.com
ACME_EMAIL=admin@yourdomain.com

# Optional: Service Overrides
JELLYFIN_PORT=8096
SONARR_PORT=8989
RADARR_PORT=7878

Service-Specific Configuration

Reverse Proxy Setup

Traefik Configuration:

# Traefik automatically configured with labels
# Custom configuration in ~/hops/config/traefik/

# Example dynamic configuration
mkdir -p ~/hops/config/traefik/dynamic
cat > ~/hops/config/traefik/dynamic/middleware.yml << 'EOF'
http:
  middlewares:
    default-headers:
      headers:
        frameDeny: true
        sslRedirect: true
        browserXssFilter: true
        contentTypeNosniff: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsPreload: true
        stsSeconds: 31536000
EOF

Nginx Proxy Manager:

• Access admin interface on port 81
• Default credentials: admin@example.com / changeme
• Configure SSL certificates through web interface

Caddy Configuration:

# Create Caddy configuration directory
mkdir -p ~/hops/config/caddy

# Create custom Caddyfile
cat > ~/hops/config/caddy/Caddyfile << 'EOF'
# Global options
{
    email your-email@domain.com
    # Optional: Use custom CA
    # acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}

# Main domain
yourdomain.com {
    reverse_proxy jellyfin:8096
    
    # Custom headers
    header {
        # Security headers
        Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
        X-Content-Type-Options "nosniff"
        X-Frame-Options "DENY"
        Referrer-Policy "strict-origin-when-cross-origin"
    }
}

# Subdomain routing
sonarr.yourdomain.com {
    reverse_proxy sonarr:8989
}

radarr.yourdomain.com {
    reverse_proxy radarr:7878
}

# Internal services (authentication required)
portainer.yourdomain.com {
    reverse_proxy portainer:9000
    
    # Optional: IP allowlist
    @internal {
        remote_ip 192.168.1.0/24 10.0.0.0/8
    }
    handle @internal {
        reverse_proxy portainer:9000
    }
    handle {
        respond "Access denied" 403
    }
}
EOF

Authelia Integration

# Authelia configuration
mkdir -p ~/hops/config/authelia

# Example configuration (simplified)
cat > ~/hops/config/authelia/configuration.yml << 'EOF'
theme: dark
default_redirection_url: https://yourdomain.com

server:
  host: 0.0.0.0
  port: 9091

log:
  level: warn

authentication_backend:
  file:
    path: /config/users_database.yml
    password:
      algorithm: argon2id

access_control:
  default_policy: deny
  rules:
    - domain: jellyfin.yourdomain.com
      policy: bypass
    - domain: "*.yourdomain.com"
      policy: one_factor

session:
  name: authelia_session
  domain: yourdomain.com
  
regulation:
  max_retries: 3
  ban_time: 10m

storage:
  local:
    path: /config/db.sqlite3

notifier:
  filesystem:
    filename: /config/notification.txt
EOF

Service Management Commands

User Operations Script (No Sudo Required)

# Service status and control
./user-operations status              # View all service status
./user-operations status jellyfin     # View specific service
./user-operations logs jellyfin       # View service logs
./user-operations logs jellyfin -f    # Follow logs

# Deployment operations
./user-operations deploy              # Deploy all services
./user-operations stop               # Stop all services
./user-operations restart            # Restart all services
./user-operations restart jellyfin   # Restart specific service

# Update operations
./user-operations update             # Update all containers
./user-operations update jellyfin    # Update specific container

Direct Docker Compose Commands

cd ~/hops

# Service management
docker compose ps                           # List services
docker compose up -d                        # Start all services
docker compose down                         # Stop all services
docker compose restart                      # Restart all services
docker compose restart jellyfin             # Restart specific service

# Logs and monitoring
docker compose logs                          # View all logs
docker compose logs -f jellyfin             # Follow specific service logs
docker compose logs --tail=100 sonarr       # Last 100 lines

# Updates and maintenance
docker compose pull                          # Pull new images
docker compose up -d --force-recreate       # Recreate containers
docker compose down && docker compose up -d # Full restart

# Resource monitoring
docker stats                                 # Real-time resource usage
docker system df                            # Disk usage
docker system prune                         # Clean unused data

New Modular Architecture

HOPS v3.1.0+ introduces a modular library system:

hops/
├── lib/                    # Shared libraries
│   ├── common.sh          # Logging, UI, utilities
│   ├── system.sh          # OS detection, requirements
│   ├── docker.sh          # Docker operations
│   ├── security.sh        # Security functions
│   ├── validation.sh      # Input validation
│   ├── secrets.sh         # Secret management
│   └── privileges.sh      # Privilege separation
├── setup                  # Main installation wrapper
├── privileged-setup       # Root-only operations
├── user-operations        # User operations
├── services-improved      # Enhanced service definitions
└── hops                  # Main script

Using Library Functions

# Source libraries in custom scripts
source lib/common.sh
source lib/system.sh

# Use logging functions
log "INFO" "Starting custom operation"
warning "This is a warning message"
error_exit "Fatal error occurred"

# Use system functions
detect_os
check_system_requirements 2 10  # 2GB RAM, 10GB disk

# Use validation functions
validate_port "8080"
validate_domain "example.com"

🔒 Security Features & Hardening

Automatic Security Hardening

HOPS automatically implements several security measures:

Firewall Configuration (Linux)

# UFW rules automatically created
sudo ufw status

# Manual firewall management
sudo ufw allow 8096/tcp comment "Jellyfin"
sudo ufw allow 9000/tcp comment "Portainer"
sudo ufw delete allow 8096/tcp  # Remove rule

File Permissions

# Automatic permission hardening
# Secrets: 600 (owner read/write only)
# Configs: 644 (owner write, group/other read)
# Scripts: 755 (executable)

# Manual permission fixes
chmod 600 ~/hops/.env
chmod 644 ~/hops/docker-compose.yml
chmod 755 ~/hops/user-operations

Network Isolation

# Docker network isolation
docker network ls | grep homelab
docker network inspect homelab

# View network configuration
docker compose config | grep network

Security Auditing

# Run security audit
./lib/security.sh audit

# Check for security issues
./lib/security.sh check-passwords
./lib/security.sh check-permissions
./lib/security.sh check-firewall

# Security recommendations
./lib/security.sh recommendations

SSL/TLS Configuration

Traefik SSL

Traefik automatically handles SSL certificates with Let's Encrypt:

# Check certificate status
docker compose logs traefik | grep -i certificate

# Manual certificate renewal (if needed)
docker compose restart traefik

Custom SSL Certificates

# For custom certificates, place in:
# ~/hops/config/traefik/certs/
# - yourdomain.com.crt
# - yourdomain.com.key

# Update Traefik configuration to use custom certs

🆘 Troubleshooting

Common Issues & Solutions

Docker Issues

Docker Not Starting:

# Check Docker status
systemctl status docker

# Restart Docker
sudo systemctl restart docker

# Check Docker logs
journalctl -u docker --since "1 hour ago"

# Reinstall Docker (Linux)
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
newgrp docker

Docker Compose Errors:

# Validate compose file
docker compose config

# Check for syntax errors
docker compose config --quiet

# Force recreate containers
docker compose up -d --force-recreate

Service-Specific Issues

Service Won't Start:

# Check service logs
docker compose logs [service-name]

# Check container status
docker compose ps

# Restart service
docker compose restart [service-name]

# Check port conflicts
sudo lsof -i :[port-number]

Permission Issues:

# Fix ownership (Linux)
sudo chown -R $USER:$USER /opt/appdata
sudo chown -R $USER:$USER /mnt/media
sudo chown -R $USER:$USER ~/hops

# Fix ownership (macOS)
sudo chown -R $USER:$USER ~/hops/config
sudo chown -R $USER:$USER ~/hops/media

# Check PUID/PGID values
id $USER  # Should match PUID/PGID in .env

Database Issues:

# Reset service database (example: Sonarr)
docker compose down sonarr
rm -rf ~/hops/config/sonarr/sonarr.db*  # macOS
rm -rf /opt/appdata/sonarr/sonarr.db*   # Linux
docker compose up -d sonarr

Network Issues

Can't Access Services:

# Check if services are running
docker compose ps

# Check port mapping
docker compose port jellyfin 8096

# Check firewall (Linux)
sudo ufw status

# Check local firewall (macOS)
# System Preferences → Security & Privacy → Firewall

# Find container IP
docker inspect jellyfin | grep IPAddress

Reverse Proxy Issues:

# Check proxy logs
docker compose logs traefik
docker compose logs nginx-proxy-manager

# Verify DNS resolution
nslookup yourdomain.com

# Check certificate status
openssl s_client -connect yourdomain.com:443 -servername yourdomain.com

Platform-Specific Issues

macOS Issues:

# Docker Desktop not starting
open /Applications/Docker.app

# Homebrew issues
brew doctor
brew update && brew upgrade

# Fix keychain authentication
security unlock-keychain ~/Library/Keychains/login.keychain

Windows/WSL2 Issues:

# WSL2 not starting
wsl --shutdown
wsl -d Ubuntu-22.04

# Docker Desktop WSL2 integration
# Check Docker Desktop → Settings → Resources → WSL Integration

# File permission issues
# Ensure files are in WSL2 filesystem, not /mnt/c/

Log Analysis

Log Locations

• Installation Logs:
  • Linux: /var/log/hops/
  • macOS: /usr/local/var/log/hops/
  • Windows: ~/hops/logs/
• Service Logs: docker compose logs [service-name]
• System Logs: journalctl -u docker

Log Analysis Commands

# HOPS installation logs
sudo tail -f /var/log/hops/hops-main-*.log

# Service logs with filtering
docker compose logs jellyfin | grep -i error
docker compose logs --since="1h" sonarr
docker compose logs --tail=100 radarr

# System resource logs
dmesg | grep -i memory
journalctl --since="1 hour ago" | grep docker

Recovery Procedures

Service Recovery

# Stop problematic service
docker compose stop [service-name]

# Remove container (keeps data)
docker compose rm [service-name]

# Recreate service
docker compose up -d [service-name]

# Full service reset (destroys data)
docker compose down [service-name]
rm -rf /path/to/service/config
docker compose up -d [service-name]

Complete System Recovery

# Stop all services
docker compose down

# Backup current state
sudo tar -czf hops-backup-$(date +%Y%m%d).tar.gz ~/hops /opt/appdata

# Clean Docker system
docker system prune -a
docker volume prune

# Restart from backup
cd ~/hops
docker compose up -d

# Or reinstall HOPS
sudo ./uninstall
sudo ./setup

📊 Performance Tuning

Resource Monitoring

# Container resource usage
docker stats

# System resource usage
htop
iotop
free -h
df -h

# Service-specific monitoring
docker compose exec portainer sh
# Access Portainer for detailed monitoring

Optimization Strategies

For Low-Resource Systems (2-4GB RAM)

# Recommended minimal stack
# - Jellyfin (media server)
# - qBittorrent (download)
# - Sonarr (TV management)
# - Portainer (monitoring)

# Resource limits in docker-compose.yml
services:
  jellyfin:
    mem_limit: 1g
    cpus: '2'
    
  sonarr:
    mem_limit: 512m
    cpus: '1'

For High-Performance Systems (8GB+ RAM)

# Full stack deployment
# Enable GPU transcoding
# Use multiple download clients
# Add monitoring stack

# GPU support (Linux only)
# Intel GPU passthrough automatically configured
devices:
  - /dev/dri:/dev/dri

Storage Optimization

# Use SSD for application data
# HDD for media storage
# Separate Docker volumes

# Example optimized storage layout
/opt/appdata -> SSD
/mnt/media   -> HDD array
~/hops       -> SSD

Update Management

# Automated updates with Watchtower
# Configure update schedule
WATCHTOWER_SCHEDULE=0 0 2 * * *  # 2 AM daily

# Manual update process
docker compose pull
docker compose up -d

# Rollback to previous version
docker compose down
docker tag service:latest service:backup
docker pull service:previous
docker compose up -d

🔧 Advanced Features

Custom Service Definitions

# Add custom services to docker-compose.yml
services:
  custom-service:
    image: custom/image:latest
    container_name: custom-service
    environment:
      - PUID=${PUID}
      - PGID=${PGID}
      - TZ=${TZ}
    volumes:
      - ${CONFIG_ROOT}/custom:/config
      - ${DATA_ROOT}:/data
    ports:
      - "8999:8999"
    restart: unless-stopped
    networks:
      - homelab

Backup Automation

# Automated backup script
#!/bin/bash
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR="/backup/hops"

# Create backup directory
mkdir -p "$BACKUP_DIR"

# Backup configurations
tar -czf "$BACKUP_DIR/config-$DATE.tar.gz" /opt/appdata

# Backup compose files
cp ~/hops/.env ~/hops/docker-compose.yml "$BACKUP_DIR/"

# Backup encrypted secrets
./lib/secrets.sh backup "$BACKUP_DIR/secrets-$DATE.enc"

# Clean old backups (keep 7 days)
find "$BACKUP_DIR" -name "*.tar.gz" -mtime +7 -delete

Development & Testing

# Development setup
git clone https://github.com/skiercm/hops.git
cd hops

# Syntax validation
bash -n lib/*.sh
bash -n *.sh

# Test service definitions
./services-improved list
./services-improved generate jellyfin

# Test installation in VM/container
# Use test environment before production deployment

---

For installation guides, see INSTALLATION.md.
For service information, see SERVICES.md.