Updates & Releases

Deploy new code releases, roll back on failure, and upgrade infrastructure services.

Checklist (Code Release)

• Pull latest code
• Build new application image
• Run database migrations
• Restart application container
• Verify health endpoint
• Monitor logs for errors

Code Release Procedure

Only the application image changes during a normal release. Infrastructure stays running.

cd /opt/zol-rag

# 1. Pull latest code
git fetch origin && git pull

# 2. Build new image
export GIT_SHA=$(git rev-parse --short HEAD)
docker build -f docker/Dockerfile.app -t zol-rag-app:${GIT_SHA} .

# 3. Run database migrations (if any)
docker run --rm \
  --network zol-network \
  --env-file .env.prod \
  zol-rag-app:${GIT_SHA} \
  alembic upgrade head

# 4. Deploy new version
APP_IMAGE=zol-rag-app:${GIT_SHA} \
  docker compose -f docker/docker-compose.app.yml --env-file .env.prod up -d

# 5. Verify health
curl -s http://localhost:80/health | python3 -m json.tool

# 6. Monitor logs for errors
docker logs zol-app --tail 100 -f

The existing scripts/deploy-prod.sh automates steps 1–6 with health checks and automatic rollback.

Rollback

If a release causes issues, rollback takes under 60 seconds (previous image is already cached).

# 1. List available images (newest first)
docker images zol-rag-app --format "{{.Tag}} {{.CreatedAt}}" | head -5

# 2. Check if migration rollback is needed
docker run --rm \
  --network zol-network \
  --env-file .env.prod \
  zol-rag-app:<previous-sha> \
  alembic history --verbose | head -10

# 3. Downgrade migrations if needed
docker run --rm \
  --network zol-network \
  --env-file .env.prod \
  zol-rag-app:${GIT_SHA} \
  alembic downgrade <target-revision>

# 4. Restart with the previous image
APP_IMAGE=zol-rag-app:<previous-sha> \
  docker compose -f docker/docker-compose.app.yml --env-file .env.prod up -d

# 5. Verify rollback
curl -s http://localhost:80/health | python3 -m json.tool

Infrastructure Upgrades

Infrastructure services (PostgreSQL, Redis, Keycloak, etc.) are upgraded independently, typically only for security patches or version bumps.

cd /opt/zol-rag

# 1. Back up critical data first
docker exec zol-postgres pg_dump \
  -U ${POSTGRES_USER} -d ${POSTGRES_DB} \
  --format=custom --file=/tmp/backup.dump
docker cp zol-postgres:/tmp/backup.dump /backups/postgres/$(date +%Y%m%d).dump

# 2. Pull updated images
docker compose -f docker/docker-compose.infra.yml --env-file .env.prod pull

# 3. Restart infrastructure (data persists on volumes)
docker compose -f docker/docker-compose.infra.yml --env-file .env.prod up -d

# 4. Wait for all services to be healthy
docker compose -f docker/docker-compose.infra.yml --env-file .env.prod ps

Backups

PostgreSQL (CRITICAL)

# Manual backup
docker exec zol-postgres pg_dump \
  -U ${POSTGRES_USER} -d ${POSTGRES_DB} \
  --format=custom --file=/tmp/backup.dump
docker cp zol-postgres:/tmp/backup.dump /backups/postgres/$(date +%Y%m%d).dump

Backup Schedule

Component	Method	Frequency	Retention
PostgreSQL	pg_dump --format=custom	Daily 02:00	30 days
MinIO	mc mirror to backup location	Daily 04:00	30 days
Redis	Copy AOF file from volume	Weekly	7 days
Grafana	Dashboard JSON export	On change	Git-tracked

Stopping Services

# Stop application only (keeps infrastructure running)
docker compose -f docker/docker-compose.app.yml down

# Stop infrastructure (data persists on volumes)
docker compose -f docker/docker-compose.infra.yml --env-file .env.prod down

# DANGER: Stop and DELETE all data volumes
docker compose -f docker/docker-compose.infra.yml --env-file .env.prod down -v

Cleaning Up Old Images

# Remove unused images
docker image prune -f

# Remove build cache
docker builder prune -f

# Check disk usage
docker system df

Next: Troubleshooting →