BuildKite: Set environment variables for downstream steps

I would like to export a job input parameters as environment variables to the subsequent steps.

According to the documentation, I can use env set:

buildkite-agent env set [variable]

Sets environment variables in the current job execution environment. Changes to the job environment variables only apply to subsequent phases of the job.

However it does not seem to work.

Simplified example:

env:
    DEFAULT_FOO: a_foo

name: "Parametrized test"
steps:
- input: "Parameters"
  label: "Set parameters"
  key: "input-params"
  fields:
    - text: "Name of Foo"
      key: "foo_param"
      hint: "Example: Foo, foo, FOO, ..."
      required: true
      default: "${DEFAULT_FOO}"
- command: |
    echo FOO="$(buildkite-agent meta-data get foo_param)"
    buildkite-agent env set FOO="$(buildkite-agent meta-data get foo_param)"
    buildkite-agent pipeline upload .buildkite/pipeline.matrix.yml

  depends_on: "input-params"
  label: "Configure matrix pipeline"

I can see meta-data get works as expected. I.e. this line:

echo FOO="$(buildkite-agent meta-data get foo_param)"

Prints FOO=<whatever value I set in input>. As one would expect.

However this step fails:

buildkite-agent env set FOO="$(buildkite-agent meta-data get foo_param)"

With error:

Could not create Job API client: BUILDKITE_AGENT_JOB_API_SOCKET empty or undefined
This command can only be used from hooks or plugins running under a job executor
where the "job-api" experiment is enabled.

Am I doing something wrong? Is this a bug? Is the documentation misleading?

Would also appreciate any suggestion for alternative ways to pass input parameters to downstream pipelines.

Solution

Answering with help from BK team on this GH issue.

Could not create Job API client: BUILDKITE_AGENT_JOB_API_SOCKET empty or undefined

This error is due to the (experimental) Job API being off by default. To enable, add experiment=job-api in buildkite-agent.conf

However, using the JOB API was the wrong approach for this problem.

All I needed was an export, i.e.:

export echo FOO="$(buildkite-agent meta-data get foo_param)"

In the same step, I can reference and use FOO, but needs to be double-escaped, i.e.:

echo "The value of FOO is $$FOO"

But in subsequent steps (loaded from .buildkite/pipeline.matrix.yml in my case), there's no need to double-escape. i.e.:

echo "The value of FOO is $FOO"