python-3.xdictionaryfilterunique

Python, filter unique but keep all columns

I am trying to filter out unique matches, on a dict with hash tables in. I have created a filter column. that is a hash of DST+PORT+PROT+SRC. So I just want to filter on the hash. e.g

[{'DST': '192.168.171.255',
  'FILTER': 4044338851943978111,
  'PORT': 137,
  'PROTO': 17,
  'SRC': '192.168.171.161'},
 {'DST': '192.168.171.255',
  'FILTER': 4044338851943978111,
  'PORT': 137,
  'PROTO': 17,
  'SRC': '192.168.171.161'},
 {'DST': '192.168.171.255',
  'FILTER': 4044338851943978111,
  'PORT': 137,
  'PROTO': 17,
  'SRC': '192.168.171.161'},
 {'DST': '192.168.171.255',
  'FILTER': 4044338851943978111,
  'PORT': 137,
  'PROTO': 17,
  'SRC': '192.168.171.161'},
 {'DST': '192.168.171.69',
  'FILTER': 1048673258922045509,
  'PORT': 63971,
  'PROTO': 6,
  'SRC': '35.157.63.228'}
]

I want the output to be

[ {'DST': '192.168.171.255',
  'FILTER': 4044338851943978111,
  'PORT': 137,
  'PROTO': 17,
  'SRC': '192.168.171.161'},
 {'DST': '192.168.171.69',
  'FILTER': 1048673258922045509,
  'PORT': 63971,
  'PROTO': 6,
  'SRC': '35.157.63.228'}
]

Thanks

Solution

  • Dictionary keys must be unique so:

    unique_data = list({entry['FILTER']: entry for entry in data}.values())

    will remove duplicates and give you a list in the format you need.