I succeed Loki's "SimpleScalable" deployment mode using S3 as storage. Now I have cleaned S3 buckets, and hope to try "Distributed" mode.
Based on
I wrote my deployment code:
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: production-hm-loki
namespace: production-hm-argo-cd
labels:
app.kubernetes.io/name: hm-loki
spec:
project: production-hm
source:
repoURL: https://grafana.github.io/helm-charts
# https://artifacthub.io/packages/helm/grafana/loki
targetRevision: 0.80.1
chart: loki-distributed
helm:
releaseName: hm-loki
values: |
---
deploymentMode: Distributed
serviceAccount:
create: true
name: hm-loki
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::272394222652:role/LokiRole-hm-loki
loki:
auth_enabled: true
schemaConfig:
configs:
- from: 2025-01-01
store: tsdb
object_store: s3
schema: v13
index:
prefix: loki_index_
period: 24h
storage_config:
aws:
region: us-west-2
bucketnames: production-hm-loki-chunk-bucket
s3forcepathstyle: false
boltdb_shipper:
shared_store: s3
compactor:
shared_store: s3
pattern_ingester:
enabled: true
ingester:
chunk_encoding: zstd
max_chunk_age: 24h
limits_config:
# https://grafana.com/docs/grafana/latest/explore/simplified-exploration/logs/access/
allow_structured_metadata: true
volume_enabled: true
querier:
max_concurrent: 4
compactor:
retention_enabled: true
delete_request_store: s3
ruler:
enable_api: true
storage:
type: s3
s3:
region: us-west-2
bucketnames: production-hm-loki-ruler-bucket
s3forcepathstyle: false
storage:
type: s3
s3:
region: us-west-2
bucketNames:
chunks: production-hm-loki-chunk-bucket
ruler: production-hm-loki-ruler-bucket
admin: production-hm-loki-admin-bucket
distributor:
replicas: 3
maxUnavailable: 1
ingester:
replicas: 3
maxUnavailable: 2
zoneAwareReplication:
enabled: false
querier:
replicas: 3
maxUnavailable: 2
queryFrontend:
replicas: 2
maxUnavailable: 1
queryScheduler:
replicas: 2
compactor:
replicas: 1
indexGateway:
replicas: 2
maxUnavailable: 1
bloomPlanner:
replicas: 0
bloomBuilder:
replicas: 0
bloomGateway:
replicas: 0
minio:
enabled: false
# Zero out replica counts of monolithic deployment mode
singleBinary:
replicas: 0
# Zero out replica counts of simple scalable deployment mode
backend:
replicas: 0
read:
replicas: 0
write:
replicas: 0
destination:
namespace: production-hm-loki
server: https://kubernetes.default.svc
syncPolicy:
syncOptions:
- ServerSideApply=true
automated:
prune: true
However, this gives me error
at least one bucket name must be specified
github.com/grafana/loki/pkg/storage/chunk/client/aws.buckets
/src/loki/pkg/storage/chunk/client/aws/s3_storage_client.go:335
github.com/grafana/loki/pkg/storage/chunk/client/aws.NewS3ObjectClient
/src/loki/pkg/storage/chunk/client/aws/s3_storage_client.go:154
github.com/grafana/loki/pkg/storage.NewObjectClient
/src/loki/pkg/storage/factory.go:621
github.com/grafana/loki/pkg/storage.NewChunkClient
/src/loki/pkg/storage/factory.go:451
github.com/grafana/loki/pkg/storage.(*store).chunkClientForPeriod
/src/loki/pkg/storage/store.go:200
github.com/grafana/loki/pkg/storage.(*store).init
/src/loki/pkg/storage/store.go:164
github.com/grafana/loki/pkg/storage.NewStore
/src/loki/pkg/storage/store.go:155
github.com/grafana/loki/pkg/loki.(*Loki).initStore
/src/loki/pkg/loki/modules.go:690
github.com/grafana/dskit/modules.(*Manager).initModule
/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:136
github.com/grafana/dskit/modules.(*Manager).InitModuleServices
/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108
github.com/grafana/loki/pkg/loki.(*Loki).Run
/src/loki/pkg/loki/loki.go:461
main.main
/src/loki/cmd/loki/main.go:110
runtime.main
/usr/local/go/src/runtime/proc.go:271
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1695
error creating object client
github.com/grafana/loki/pkg/storage.(*store).chunkClientForPeriod
/src/loki/pkg/storage/store.go:202
github.com/grafana/loki/pkg/storage.(*store).init
/src/loki/pkg/storage/store.go:164
github.com/grafana/loki/pkg/storage.NewStore
/src/loki/pkg/storage/store.go:155
github.com/grafana/loki/pkg/loki.(*Loki).initStore
/src/loki/pkg/loki/modules.go:690
github.com/grafana/dskit/modules.(*Manager).initModule
/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:136
github.com/grafana/dskit/modules.(*Manager).InitModuleServices
/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108
github.com/grafana/loki/pkg/loki.(*Loki).Run
/src/loki/pkg/loki/loki.go:461
main.main
/src/loki/cmd/loki/main.go:110
runtime.main
/usr/local/go/src/runtime/proc.go:271
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1695
error initialising module: store
github.com/grafana/dskit/modules.(*Manager).initModule
/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138
github.com/grafana/dskit/modules.(*Manager).InitModuleServices
/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108
github.com/grafana/loki/pkg/loki.(*Loki).Run
/src/loki/pkg/loki/loki.go:461
main.main
/src/loki/cmd/loki/main.go:110
runtime.main
/usr/local/go/src/runtime/proc.go:271
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1695
level=warn ts=2025-02-17T23:36:30.768798099Z caller=loki.go:288 msg="global timeout not configured, using default engine timeout (\"5m0s\"). This behavior will change in the next major to always use the default global timeout (\"5m\")."
level=info ts=2025-02-17T23:36:30.770080649Z caller=main.go:108 msg="Starting Loki" version="(version=2.9.10, branch=HEAD, revision=7664eda07b)"
level=info ts=2025-02-17T23:36:30.770488633Z caller=server.go:322 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2025-02-17T23:36:30.771113623Z caller=memberlist_client.go:434 msg="Using memberlist cluster label and node name" cluster_label=hm-loki.production-hm-loki node=hm-loki-loki-distributed-ingester-2-36639255
level=info ts=2025-02-17T23:36:30.771130514Z caller=modules.go:1342 msg="failed to initialize usage report" err="at least one bucket name must be specified"
level=warn ts=2025-02-17T23:36:30.771151019Z caller=experimental.go:20 msg="experimental feature in use" feature="In-memory (FIFO) cache - embedded-cache"
level=warn ts=2025-02-17T23:36:30.771210656Z caller=cache.go:127 msg="fifocache config is deprecated. use embedded-cache instead"
level=warn ts=2025-02-17T23:36:30.771216642Z caller=experimental.go:20 msg="experimental feature in use" feature="In-memory (FIFO) cache - chunksembedded-cache"
level=error ts=2025-02-17T23:36:30.771357894Z caller=log.go:230 msg="error running loki" err="at least one bucket name must be specified\ngithub.com/grafana/loki/pkg/storage/chunk/client/aws.buckets\n\t/src/loki/pkg/storage/chunk/client/aws/s3_storage_client.go:335\ngithub.com/grafana/loki/pkg/storage/chunk/client/aws.NewS3ObjectClient\n\t/src/loki/pkg/storage/chunk/client/aws/s3_storage_client.go:154\ngithub.com/grafana/loki/pkg/storage.NewObjectClient\n\t/src/loki/pkg/storage/factory.go:621\ngithub.com/grafana/loki/pkg/storage.NewChunkClient\n\t/src/loki/pkg/storage/factory.go:451\ngithub.com/grafana/loki/pkg/storage.(*store).chunkClientForPeriod\n\t/src/loki/pkg/storage/store.go:200\ngithub.com/grafana/loki/pkg/storage.(*store).init\n\t/src/loki/pkg/storage/store.go:164\ngithub.com/grafana/loki/pkg/storage.NewStore\n\t/src/loki/pkg/storage/store.go:155\ngithub.com/grafana/loki/pkg/loki.(*Loki).initStore\n\t/src/loki/pkg/loki/modules.go:690\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:136\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:461\nmain.main\n\t/src/loki/cmd/loki/main.go:110\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:271\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nerror creating object client\ngithub.com/grafana/loki/pkg/storage.(*store).chunkClientForPeriod\n\t/src/loki/pkg/storage/store.go:202\ngithub.com/grafana/loki/pkg/storage.(*store).init\n\t/src/loki/pkg/storage/store.go:164\ngithub.com/grafana/loki/pkg/storage.NewStore\n\t/src/loki/pkg/storage/store.go:155\ngithub.com/grafana/loki/pkg/loki.(*Loki).initStore\n\t/src/loki/pkg/loki/modules.go:690\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:136\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:461\nmain.main\n\t/src/loki/cmd/loki/main.go:110\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:271\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nerror initialising module: store\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:461\nmain.main\n\t/src/loki/cmd/loki/main.go:110\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:271\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"
Any guide would be appreciate, thanks!
I did more research:
After I switch from loki-distributed to loki Helm chart and using deploymentMode: Distributed, it just works well. Here is my current deployment code:
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: production-hm-loki
namespace: production-hm-argo-cd
labels:
app.kubernetes.io/name: hm-loki
spec:
project: production-hm
source:
repoURL: https://grafana.github.io/helm-charts
# https://artifacthub.io/packages/helm/grafana/loki
targetRevision: 6.27.0
chart: loki
helm:
releaseName: hm-loki
values: |
# https://github.com/grafana/loki/blob/main/production/helm/loki/values.yaml
# https://github.com/grafana/loki/blob/main/production/helm/loki/distributed-values.yaml
# https://grafana.com/docs/loki/latest/setup/install/helm/install-microservices/
---
deploymentMode: Distributed
serviceAccount:
create: true
name: hm-loki
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::272394222652:role/LokiRole-hm-loki
loki:
auth_enabled: true
schemaConfig:
configs:
- from: 2025-01-01
store: tsdb
object_store: s3
schema: v13
index:
prefix: loki_index_
period: 24h
storage_config:
aws:
region: us-west-2
bucketnames: production-hm-loki-chunk-bucket
s3forcepathstyle: false
pattern_ingester:
enabled: true
ingester:
chunk_encoding: zstd
max_chunk_age: 24h
# https://github.com/grafana/loki/issues/8615#issuecomment-1741827915
autoforget_unhealthy: true
limits_config:
# https://grafana.com/docs/loki/latest/operations/automatic-stream-sharding/
shard_streams:
enabled: true
# https://grafana.com/docs/grafana/latest/explore/simplified-exploration/logs/access/
allow_structured_metadata: true
volume_enabled: true
# https://grafana.com/docs/loki/latest/operations/storage/retention/
retention_period: 744h
# https://grafana.com/docs/loki/latest/operations/request-validation-rate-limits/
ingestion_rate_mb: 50
ingestion_burst_size_mb: 100
per_stream_rate_limit: 50MB
per_stream_rate_limit_burst: 100MB
query_scheduler:
max_outstanding_requests_per_tenant: 32768
querier:
max_concurrent: 16
server:
# 128 MiB
grpc_server_max_recv_msg_size: 134217728
# 128 MiB
grpc_server_max_send_msg_size: 134217728
compactor:
retention_enabled: true
delete_request_store: s3
ruler:
enable_api: true
storage:
type: s3
s3:
region: us-west-2
bucketnames: production-hm-loki-ruler-bucket
s3forcepathstyle: false
alertmanager_url: http://hm-prometheus-kube-pr-alertmanager.production-hm-prometheus:9093
storage:
type: s3
s3:
region: us-west-2
bucketNames:
chunks: production-hm-loki-chunk-bucket
ruler: production-hm-loki-ruler-bucket
admin: production-hm-loki-admin-bucket
gateway:
enabled: true
replicas: 3
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 200m
memory: 256Mi
indexGateway:
replicas: 3
maxUnavailable: 1
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 200m
memory: 2Gi
distributor:
autoscaling:
enabled: true
minReplicas: 6
maxReplicas: 15
targetCPUUtilizationPercentage: 80
targetMemoryUtilizationPercentage: 80
resources:
requests:
cpu: 1000m
memory: 256Mi
limits:
cpu: 8000m
memory: 8Gi
ingester:
zoneAwareReplication:
enabled: true
autoscaling:
enabled: true
minReplicas: 3
maxReplicas: 15
targetCPUUtilizationPercentage: 80
targetMemoryUtilizationPercentage: 80
resources:
requests:
cpu: 2000m
memory: 2Gi
limits:
cpu: 8000m
memory: 8Gi
queryFrontend:
autoscaling:
enabled: true
minReplicas: 3
maxReplicas: 6
targetCPUUtilizationPercentage: 80
targetMemoryUtilizationPercentage: 80
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 200m
memory: 1Gi
queryScheduler:
replicas: 3
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 500m
memory: 2Gi
querier:
autoscaling:
enabled: true
minReplicas: 6
maxReplicas: 15
targetCPUUtilizationPercentage: 80
targetMemoryUtilizationPercentage: 80
resources:
requests:
cpu: 2000m
memory: 1Gi
limits:
cpu: 4000m
memory: 2Gi
compactor:
replicas: 3
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 500m
memory: 2Gi
chunksCache:
replicas: 3
resources:
requests:
cpu: 100m
memory: 1Gi
limits:
cpu: 200m
memory: 8Gi
memcachedExporter:
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 200m
memory: 256Mi
lokiCanary:
enabled: true
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 200m
memory: 256Mi
bloomPlanner:
replicas: 0
bloomBuilder:
replicas: 0
bloomGateway:
replicas: 0
minio:
enabled: false
# Zero out replica counts of monolithic deployment mode
singleBinary:
replicas: 0
# Zero out replica counts of simple scalable deployment mode
backend:
replicas: 0
read:
replicas: 0
write:
replicas: 0
destination:
namespace: production-hm-loki
server: https://kubernetes.default.svc
syncPolicy:
syncOptions:
- ServerSideApply=true
automated:
prune: true