[SOLVED] how to use jq to filter select items not in list?

how to use jq to filter select items not in list?

In jq, I can select an item in a list fairly easily:

$ echo '["a","b","c","d","e"]' | jq '.[] | select(. == ("a","c"))'

Or if you prefer to get it as an array:

$ echo '["a","b","c","d","e"]' | jq 'map(select(. == ("a","c")))'

But how do I select all of the items that are not in the list? Certainly . != ("a","c") does not work:

$ echo '["a","b","c","d","e"]' | jq 'map(select(. != ("a","c")))'
[
  "a",
  "b",
  "b",
  "c",
  "d",
  "d",
  "e",
  "e"
]

The above gives every item twice, except for "a" and "c"

Same for:

$ echo '["a","b","c","d","e"]' | jq '.[] | select(. != ("a","c"))'
"a"
"b"
"b"
"c"
"d"
"d"
"e"
"e"

How do I filter out the matching items?

Solution

The simplest and most robust (w.r.t. jq versions) approach would be to use the builtin -:

$ echo '["a","b","c","d","e"]' | jq -c '. - ["a","c"]'
["b","d","e"]

If the blacklist is very long and riddled with duplicates, then it might be appropriate to remove them (e.g. with unique).

The problem can also be solved (in jq 1.4 and up) using index and not, e.g.

["a","c"] as $blacklist
| .[] | select( . as $in | $blacklist | index($in) | not)

Or, with a variable passed in from the command-line (jq --argjson blacklist ...):

.[] | select( . as $in | $blacklist | index($in) | not)

To preserve the list structure, one can use map( select( ...) ).

With jq 1.5 or later, you could also use any or all, e.g.

def except(blacklist):
  map( select( . as $in | blacklist | all(. != $in) ) );