prometheus and cAdvisor: Error scraping target: label_value_length_limit exceeded

I'm using Prometheus 3.4.1, set up as described here. I run it and cAdvisor this way:

docker run -d --name prometheus -p 9090:9090 --group-add 988 \
-v "/etc/$p_conf":"/etc/prometheus/$p_conf" \
-v "$pv":/prometheus -v /var/run/docker.sock:/var/run/docker.sock \
prom/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/prometheus \
--web.console.libraries=/etc/prometheus/console_libraries \
--web.console.templates=/etc/prometheus/consoles \
--web.enable-lifecycle

docker run \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:ro \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --publish=8080:8080 \
  --detach=true \
  --name=cadvisor \
  gcr.io/cadvisor/cadvisor:latest

The related part of prometheus config is:

  - job_name: 'cadvisor'
    static_configs:
      - targets: ['localhost:8080']

In targets I see error:

Error scraping target: label_value_length_limit exceeded (metric: container_fs_inodes_free, label name: device, value: "/var/lib/docker/overlay2/34...20", length: 203, limit: 200)

As far as I understood, neither metric_relabel_configs nor relabel_configs will work, please correct me if I'm wrong. I tried to fix the issue by adding

--storage.tsdb.label_value_length_limit=204

to the prometeus commands but got error:

unknown long flag '--storage.tsdb.label_value_length_limit'

My guess is, it's because that option was valid for 2.x. What is the newer version of this option? How can I fix that?

If you grep Prometheus Configuration, there are 2 places where you can limit the label_value_length:

1. global
2. scrape_config

Alternatively, you could relabel values potentially removing the static prefix (/var/lib/docker/overlay2):

You can use metric_relabel_config to process values that are longer than your label_value_length. For example to extract a value from within the overlay that's unique. You will want to retain a relabel_config that e.g trims any values that exceed your limit:

global:
  # Setting an arbitrarily lower (!) limit to prove the point 
  label_value_length_limit: 150

scrape_configs:
- job_name: cadvisor
  metric_relabel_configs:
    # Create a label "foo" by extracting a hex string value from "device" value
  - source_labels: [device]
    target_label: foo
    # Replace "x" with a numerical value corresponding to the length
    regex: "/var/lib/docker/overlay2/([0-9a-f]{x}/.+"
    replacement: "${1}"
    # Trim any "device" values that exceed the limit
  - source_labels: [device]
    target_label: device
    regex: "^(.{150}).+$"
    replacement: "$1"
  static_configs:
  - targets:
    - "localhost:8080"