bashrecursionyaml

Merging yaml config files recursively with bash

Is it possible using some smart piping and coding, to merge yaml files recursively? In PHP, I make an array of them (each module can add or update config nodes of/in the system).

The goal is an export shellscript that will merge all separate module folders' config files into big merged files. It's faster, efficient, and the customer does not need the modularity at the time we deploy new versions via FTP, for example.

It should behave like the PHP function: array_merge_recursive

The filesystem structure is like this:

mod/a/config/sys.yml
mod/a/config/another.yml
mod/b/config/sys.yml
mod/b/config/another.yml
mod/c/config/totally-new.yml
sys/config/sys.yml

Config looks like:

date:
   format:
      date_regular: %d-%m-%Y

And a module may, say, do this:

date:
   format:
      date_regular: regular dates are boring
      date_special: !!!%d-%m-%Y!!!

So far, I have:

#!/bin/bash
#........
cp -R $dir_project/ $dir_to/
for i in $dir_project/mod/*/
do
    cp -R "${i}/." $dir_to/sys/
done

This of course destroys all existing config files in the loop.. (rest of the system files are uniquely named)

Basically, I need a yaml parser for the command line, and an array_merge_recursive like alternative. Then a yaml writer to ouput it merged. I fear I have to start to learn Python because bash won't cut it on this one.

Solution

  • You can use for example perl. The next oneliner:

    perl -MYAML::Merge::Simple=merge_files -MYAML -E 'say Dump merge_files(@ARGV)' f1.yaml f2.yaml

    for the next input files: f1.yaml

    date:
  epoch: 2342342343
  format:
    date_regular: "%d-%m-%Y"

    f2.yaml

    date:
  format:
    date_regular: regular dates are boring
    date_special: "!!!%d-%m-%Y!!!"

    prints the merged result...

    ---
date:
  epoch: 2342342343
  format:
    date_regular: regular dates are boring
    date_special: '!!!%d-%m-%Y!!!'

    Because @Caleb pointed out that the module now is develeloper only, here is an replacement. It is a bit longer and uses two (but commonly available) modules:

    perl -MYAML=LoadFile,Dump -MHash::Merge::Simple=merge -E 'say Dump(merge(map{LoadFile($_)}@ARGV))' f1.yaml f2.yaml

    produces the same as above.