yamljqyq

Combine JSON to YAML list


In a GitLab pipeline job I have multiple JSON files. I need these JSON files as YAML and I would like to have them in the same YAML file as a list. They do not always have the same keys.

I would like the following structure so that I later can iterate over the "mystuff" list with the keys being the filenames (unique)

mystuff:
  filename:
    content: <whatever>
  filename2:
    content: <whatever>

I have got this far with yq:

yq eval-all -o=yaml -P '
  {
    "mystuff": (
      . as $doc
      | {
          (filename
            | split("/")[-1]
            | sub("\\.json$"; "")
          ): $doc
        }
    )
  }
' "${FILES[@]}" > "$DASH_DIR/_mystuff.yaml"

which gives me:

mystuff:
  filename:
    content: <whatever>
----
mystuff:
  filename2:
    content: <whatever>

I can't figure out how to get it in a single list without changing or merging content or the previous code breaking.

My current solution to this is to run an awk command after the yq command that removes all "----" and every mystuff after the first.

EDIT: Adding some basic JSON due to request, the real ones are 300-800 line with more objects and lists

{
    "displayName": "Basic JSON 1",
    "dFilters": [],
    "description": "",
    "labels": {
        "overview": "",
        "specific": "",
        "team": ""
    },
    "layout": {}
}
{
    "displayName": "Basic JSON 2",
    "cFilters": [],
    "description": "",
    "labels": {
        "overview": "",
        "specific": "",
        "team": ""
    },
    "map": {}
}

My goal of these 2 files would be 1 YAML file _result.yaml

result:
  nameOfTheJsonFile1:
    displayName: Basic JSON 1
    dFilters: []
    description: ''
    labels:
      overview: ''
      specific: ''
      team: ''
    layout: {}
  nameOfTheJsonFile2:
    displayName: Basic JSON 2
    cFilters: []
    description: ''
    labels:
      overview: ''
      specific: ''
      team: ''
    map: {}

Solution

  • You could use ireduce to successively build up the result object:

    yq ea -oy '
      {filename: .} as $doc ireduce ({}; . + $doc) | {"mystuff": .}
    ' "${files[@]}"
    

    This would prints something along the lines of

    mystuff:
      file1.json:
        first: content
      file2.json:
        second: content
    

    Tested with mikefarah/yq version v4.47.1