I'm deploying Loki with the helm chart loki-distributed on our Kubernetes cluster, and trying to configure Loki to use an s3 bucket for chunks and indexes, but no folder or file has been created on my s3 bucket, here is my values.yaml file used for the chart:
loki:
ingester:
# Disable chunk transfer which is not possible with statefulsets
# and unnecessary for boltdb-shipper
max_transfer_retries: 0
chunk_idle_period: 1h
chunk_target_size: 1536000
max_chunk_age: 1h
schema_config:
configs:
- from: 2020-05-15
# Which store to use for the index
store: boltdb-shipper
# Which store to use for the chunks
object_store: s3
schema: v12
index:
prefix: loki_
period: 24h
storage_config:
aws:
bucketnames: loki-logs-k3s
endpoint: https://s3.de.io.cloud.ovh.net/
region: de
access_key_id: "my access key id"
secret_access_key: "my secret access key"
insecure: false
sse_encryption: false
http_config:
idle_conn_timeout: 90s
response_header_timeout: 0s
insecure_skip_verify: false
boltdb-shipper:
active_index_directory: var/loki/indexes
shared_store: s3
cache_location: var/loki/cache
compactor:
working_directory: /loki/compacting
shared_store: s3
I can see in the ingester logs the following:
level=info ts=2023-08-10T08:17:39.68137629Z caller=index_set.go:107 msg="finished uploading table loki_index_19578"
level=info ts=2023-08-10T08:17:39.681383744Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table loki_index_19578"
level=info ts=2023-08-10T08:17:39.681392812Z caller=index_set.go:86 msg="uploading table loki_index_19579"
level=info ts=2023-08-10T08:17:39.681451625Z caller=table.go:318 msg="handing over indexes to shipper loki_index_19578"
level=info ts=2023-08-10T08:17:39.681460522Z caller=table.go:334 msg="finished handing over table loki_index_19578"
level=info ts=2023-08-10T08:17:39.681488176Z caller=table.go:318 msg="handing over indexes to shipper loki_index_19579"
level=info ts=2023-08-10T08:17:39.681496461Z caller=table.go:334 msg="finished handing over table loki_index_19579"
level=info ts=2023-08-10T08:17:39.686223039Z caller=index_set.go:107 msg="finished uploading table loki_index_19579"
level=info ts=2023-08-10T08:17:39.686263397Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table loki_index_19579"
In the Query frontend logs, I have this :
level=info ts=2023-08-09T15:20:37.966409453Z caller=main.go:108 msg="Starting Loki" version="(version=2.8.2, branch=HEAD, revision=9f809eda7)"
level=info ts=2023-08-09T15:20:37.966455362Z caller=modules.go:1184 msg="failed to initialize usage report" err="mkdir /var/loki: read-only file system"
level=info ts=2023-08-09T15:20:37.967143294Z caller=server.go:323 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses"
level=warn ts=2023-08-09T15:20:37.967632095Z caller=experimental.go:20 msg="experimental feature in use" feature="In-memory (FIFO) cache - frontend.embedded-cache"
level=info ts=2023-08-09T15:20:37.970551753Z caller=module_service.go:82 msg=initialising module=runtime-config
level=info ts=2023-08-09T15:20:37.970673898Z caller=module_service.go:82 msg=initialising module=server
level=info ts=2023-08-09T15:20:37.970668116Z caller=module_service.go:82 msg=initialising module=cache-generation-loader
level=info ts=2023-08-09T15:20:37.970812315Z caller=module_service.go:82 msg=initialising module=query-frontend-tripperware
level=info ts=2023-08-09T15:20:37.97084668Z caller=module_service.go:82 msg=initialising module=query-frontend
level=info ts=2023-08-09T15:20:37.971029372Z caller=loki.go:499 msg="Loki started"
Any idea ? Thank you
Well, I wasn't able to identify the issue with the configuration I provided in my question. The documentation isn't very helpful in this regard. However, I managed to make it work using the following configuration. I hope this helps.
P.S. Removing the ingester block will prevent the creation of the fake folder (chunks) and the index folder in S3, even if these values are set to their default settings in the chart. You would need to explicitly add them.
loki:
auth_enabled: false
ingester:
lifecycler:
ring:
kvstore:
store: memberlist
replication_factor: 1
chunk_idle_period: 30m
chunk_block_size: 262144
chunk_encoding: snappy
chunk_retain_period: 1m
max_transfer_retries: 0
wal:
dir: /var/loki/wal
# -- Check https://grafana.com/docs/loki/latest/configuration/#schema_config for more info on how to configure schemas
schemaConfig:
configs:
- from: "2020-10-24"
index:
period: 24h
prefix: index_
object_store: s3
schema: v11
store: boltdb-shipper
# -- Check https://grafana.com/docs/loki/latest/configuration/#storage_config for more info on how to configure storages
storageConfig:
boltdb_shipper:
active_index_directory: /var/loki/index
cache_location: /var/loki/cache
cache_ttl: 168h
shared_store: s3
filesystem:
directory: /var/loki/chunks
aws:
s3: s3://yourAccessKeyId:yourSecretKey@region/bucketName
s3forcepathstyle: true
endpoint: yourS3endppoint