In a GitLab pipeline job I have multiple JSON files. I need these JSON files as YAML and I would like to have them in the same YAML file as a list. They do not always have the same keys.
I would like the following structure so that I later can iterate over the "mystuff" list with the keys being the filenames (unique)
mystuff:
filename:
content: <whatever>
filename2:
content: <whatever>
I have got this far with yq:
yq eval-all -o=yaml -P '
{
"mystuff": (
. as $doc
| {
(filename
| split("/")[-1]
| sub("\\.json$"; "")
): $doc
}
)
}
' "${FILES[@]}" > "$DASH_DIR/_mystuff.yaml"
which gives me:
mystuff:
filename:
content: <whatever>
----
mystuff:
filename2:
content: <whatever>
I can't figure out how to get it in a single list without changing or merging content or the previous code breaking.
My current solution to this is to run an awk command after the yq command that removes all "----" and every mystuff after the first.
EDIT: Adding some basic JSON due to request, the real ones are 300-800 line with more objects and lists
{
"displayName": "Basic JSON 1",
"dFilters": [],
"description": "",
"labels": {
"overview": "",
"specific": "",
"team": ""
},
"layout": {}
}
{
"displayName": "Basic JSON 2",
"cFilters": [],
"description": "",
"labels": {
"overview": "",
"specific": "",
"team": ""
},
"map": {}
}
My goal of these 2 files would be 1 YAML file _result.yaml
result:
nameOfTheJsonFile1:
displayName: Basic JSON 1
dFilters: []
description: ''
labels:
overview: ''
specific: ''
team: ''
layout: {}
nameOfTheJsonFile2:
displayName: Basic JSON 2
cFilters: []
description: ''
labels:
overview: ''
specific: ''
team: ''
map: {}
You could use ireduce to successively build up the result object:
yq ea -oy '
{filename: .} as $doc ireduce ({}; . + $doc) | {"mystuff": .}
' "${files[@]}"
This would prints something along the lines of
mystuff:
file1.json:
first: content
file2.json:
second: content
Tested with mikefarah/yq version v4.47.1