jq

Pass parameterized functions as callbacks to `jq` function


I'd like to dynamically pass parameterized processing functions as callbacks to a selective traversing function (process_matching_and_others).

In the example below, that's based on this excellent answer to a different question, I statically define the callback functions. This works, but has some disadvantages.

Apart from lacking encapsulation in general, my concern is, that I can't move the generic traversing function (process_matching_and_others) to a library.

I understood that jq doesn't support lambdas in the way, other programming languages do.

Is there any way to achieve the goal of dynamically define callbacks when invoking the traversal function?

Or maybe a completely different approach of encapsulating the traversal while defining the processing outside the traversal function?

I thought about the option to let the traversal function output two subtrees that can then be processed separately and finally be merged. But I hope for something more straightforward and less verbose.

Here my current code and how I verify it:

$ yq -y -f demo.jq input.yaml > actual.yaml \
&& diff expected.yaml actual.yaml \
&& echo "SUCCESS" || echo "FAILURE"
# demo.jq
def callback_1(node): if node | has("x") then node.x = "changed" else . end;
def callback_2(node): if node | has("y") then del(node.y) else . end;

# This function invokes one callback for objects that are children of path_list and another callback for objects that aren't. 
# The current approach works as long as the callbacks are defined before the traverse function is defined. 
# I'd rather like to leave it to the caller of the traverse-function to decide how to process nodes that match the path and how to process others.
def process_matching_and_others(path_list):
  reduce paths(objects) as $p (.; if $p[:path_list | length] == path_list then callback_1(getpath($p)) else callback_2(getpath($p)) end)
;

process_matching_and_others(
  ["deep","path","matching"]
)

# What I'd like to do instead would be something like this: 
# process_matching_and_others(
#   ["deep","path","matching"]; callback_x; callback_y
# )
# But I don't know how to achieve something similar, because the callbacks require an argument that's passed on invocation of the callbacks by traverse_and_invoke_callbacks
# input.yaml
deep:
  path:
    matching:
      m:
        x: change it
        y: don't touch
      n:
        nn:
          x: change it
          y: don't touch
    other_a:
      a:
        x: don't touch
        y: delete it
    other_b:
      b:
        bb:
          x: don't touch
          y: delete it
    other_list:
      - a
      - b
      - c
another_other:
  - a:
      x: don't touch
      y: delete it
  - a:
      x: don't touch
      y: delete it
# expected.yaml
deep:
  path:
    matching:
      m:
        x: changed
        y: don't touch
      n:
        nn:
          x: changed
          y: don't touch
    other_a:
      a:
        x: don't touch
    other_b:
      b:
        bb:
          x: don't touch
    other_list:
      - a
      - b
      - c
another_other:
  - a:
      x: don't touch
  - a:
      x: don't touch

SUCCESS

Solution

  • Finally I found that there is an amazingly straightforward way to fulfill all requirements specified in my original post.

    Key to the solution is, that filters always receive an input (accessible as .), so it's simply not necessary to pass a parameterized function. You can pass any callback function that processes the output of the calling function:

    def process_matching_and_others(path_list; filter_for_matching; filter_for_others):
      reduce paths(objects) as $p (
        .;
        if $p[:path_list | length] == path_list
        then getpath($p) |= matching
        else getpath($p) |= others
        end
      )
    ;
    
    def callback_1: if has("x") then .x = "changed" else . end;
    def callback_2: if has("y") then del(.y) else . end;
    
    process_matching_and_others(["deep","path","matching"]; callback_1; callback_2)
    

    Or you simply pass an appropriate filter that does the processing you want. Very similar to a lambda expression in other programming languages. This is an even more dynamic approach.

    def process_matching_and_others(path_list; filter_for_matching; filter_for_others):
      reduce paths(objects) as $p (
        .;
        if $p[:path_list | length] == path_list
        then getpath($p) |= matching
        else getpath($p) |= others
        end
      )
    ;
    
    process_matching_and_others(
      ["deep","path","matching"];
      if has("x") then .x = "changed" else . end;
      if has("y") then del(.y) else . end
    )