filterffmpegdiffphotoshopdifference

difference overlay filter ffmpeg like photoshop/affinity photo

In photoshop and affinity photo there is a nice overlay filter that basically subtracts the overlaying layer from the one bellow to make a "difference map"

Example: Base image: Screenshot Mario Kart 1

Overlaying image: Screenshot Mario Kart 2

Resulting output: Diff from two screenshots

This is an example diff of overlaying a PNG with a JpegXL(I think) compressed image and then brightened: Diff generated from image compression

Location of filter in photoshop: Photoshop screenshot

Is there anyway to accomplish this with a filter in ffmpeg for an entire video?

Solution

  • Okay I found what is probbaly the best way to do it. there is an ffmpeg filter called blend which has a few modes which can work for this.

    ffmpeg -r VIDEOFPS -i VIDEO -r VIDEOFPS -i VIDEO -lavfi blend=all_mode=grainextract -c:v libx264 -crf 0 -an ./out.mp4

    grainextract will produce a grey video with only the compression artifacts, this seems to be made directly for this purpose.

    ffmpeg -r VIDEOFPS -i VIDEO -r VIDEOFPS -i VIDEO -lavfi blend=all_mode=subtract -c:v libx264 -crf 0 -an ./out.mp4

    Subtract mode will produce a green video for some reason but it will also only have the video compression.

    ffmpeg -r VIDEOFPS -i VIDEO -r VIDEOFPS -i VIDEO -lavfi blend=all_mode=difference -c:v libx264 -crf 0 -an ./out.mp4

    differenmce seems to be extremely close to subtract if not the exact same output.

    ffmpeg -r VIDEOFPS -i VIDEO -r VIDEOFPS -i VIDEO -lavfi blend=all_mode=xor -c:v libx264 -crf 0 -an ./out.mp4

    xor results in a much noiser and contrasting output but it's great for seeing exact compression artifacts in darker or less contrasting scenes

    ffmpeg -r VIDEOFPS -i VIDEO -r VIDEOFPS -i VIDEO -lavfi blend=all_mode=phoenix -c:v libx264 -crf 0 -an ./out.mp4

    I treid phoenix on a whim just because it sounded cool and it actually works for this usecase, it results in a pink video with only compression artifacts visible but a lot less defined then grainextract so it might be more accurate? idk

    Note: if you are using any recordings from OBS or Nvidia ShadowPlay you MUST specify the FPS when importing, both use variable fps to handle small hiccups and fps differences so then using the filter you will get a bunch of random 1 frame desync