Redis Data Persistence Issue in Azure Container Instance (ACI)

I am working on deploying a Redis instance in Azure Container Instances (ACI) using a custom Docker image, but I am encountering an issue with data persistence. Despite setting up proper volume mounting and configuring Redis for persistence (using the appendonly.aof file), I am still concerned about losing data during container restarts.

Here’s the setup I am using:

Dockerfile:

FROM redis:latest

# Install procps to include sysctl
RUN apt-get update && apt-get install -y procps && rm -rf /var/lib/apt/lists/*

# Add vm.overcommit_memory setting to /etc/sysctl.conf
RUN echo "vm.overcommit_memory=1" >> /etc/sysctl.conf

# Default command to run Redis with sysctl
CMD ["sh", "-c", "ulimit -n 65536 && sysctl -p && redis-server --save '300 1' --appendonly yes --dir /data --appendfilename appendonly.aof --maxmemory 3gb --maxmemory-policy noeviction"]

Terraform Configuration (ACI Deployment):

resource "azurerm_container_group" "example" {
  name                = "redis-db"
  location            = "location"
  resource_group_name = "rg"
  os_type = "Linux"

  container {
    name   = "redis-db"
    image  = "arcregistry.azurecr.io/redis:latest"
    cpu    = "1"
    memory = "6"

    ports {
      port     = 6379
      protocol = "TCP"
    }

    environment_variables = {
      "ENV"              = "prod"
      "REDIS_ULIMIT"     = "65536"
    }

    volume {
      name                = "redis-data"
      mount_path          = "/data"
      share_name          = "redisdata"
      storage_account_name = "storage_account_name"
      storage_account_key  = "storage_key"
    }

    commands = [
    "sh", "-c",
    "chmod -R 775 /data && chown -R redis:redis /data && redis-server --save '300 1' --appendonly yes --dir /data --appendfilename appendonly.aof --rdb-del-sync-files yes --maxmemory 3gb --maxmemory-policy noeviction"
]
  }

  dns_name_label = "redis-db"
  ip_address_type = "Public"
}

Even though I’ve configured Redis to use an append-only file (appendonly.aof) for data persistence and properly mounted an Azure file share to store the Redis data, I am concerned about potential data loss when the container is restarted.

• The ACI container has a volume mount configured (/data).
• The redis.conf is updated to ensure Redis writes to appendonly.aof. However, after restarting the container, I still observe that Redis does not seem to restore its data from the appendonly.aof file.

I also encountered the following warning in the Redis logs:

# WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

Is there something specific to Azure Container Instances that could be preventing data persistence in Redis?

After spending considerable time investigating this issue, I was unable to pinpoint the exact cause of the random data loss in Redis running on Azure Container Instances (ACI), even after trying different configurations of redis.conf. The problem persisted, and I couldn't achieve the stability needed for my use case.

To address this and make progress, I decided to deploy Redis on an Azure Virtual Machine (VM) within a private VNet (using terraform). This approach proved to be stable, and I was able to maintain a reliable Redis database. I chose this solution for two main reasons:

1. Cost Efficiency: Running Redis on an Azure VM allowed better cost control compared to other managed solutions.
2. Flexibility: It gave me full control to configure and manage the infrastructure according to my needs.

That said, for scenarios where quick deployment and minimal development effort are priorities, Azure Redis Cache could be a viable alternative. It offers a managed Redis service with built-in reliability and persistence, making it ideal for use cases that require minimal infrastructure management.

This solution might help others facing similar issues or looking for a balance between cost, control, and ease of use.