Intersection-over-union between two detections

I was reading through the paper : Ferrari et al. in the "Affinity Measures" section. I understood that Ferrari et al. tries to obtain affinity by :

1. Location affinity - using area of intersection-over-union between two detections
2. Appearance affinity - using Euclidean distances between Histograms
3. KLT point affinity measure

However, I have 2 main problems:

1. I cannot understand what is actually meant by intersection-over-union between 2 detections and how to calculate it
2. I tried a slightly difference appearance affinity measure. I transformed the RGB detection into HSV..concatenating the Hue and Saturation into 1 vector, and used it to compare with other detections. However, using this technique failed as a detection of a bag had a better similarity score than a detection of the same person's head (with a different orientation).

Any suggestions or solutions to my problems described above? Thank you and your help is very much appreciated.

1) You have two overlapping bounding boxes. You compute the intersection of the boxes, which is the area of the overlap. You compute the union of the overlapping boxes, which is the sum of the areas of the entire boxes minus the area of the overlap. Then you divide the intersection by the union. There is a function for that in the Computer Vision System Toolbox called bboxOverlapRatio.

2) Generally, you don't want to concatenate the color channels. What you want instead, is a 3D histogram, where the dimensions are H, S, and V.