Given a color either in RGB or Hex, how do I find colors that are similar to it? By similar I mean they should be distinguishable by small values.

The RGB color space is device dependent and not perceptually uniform. If you intend to measure color similarities, you should transform first the RGB values into a device-independent and more perceptually uniform color space, for example CIELAB. Then you can measure the color differences through a proper similarity metric, like Lab Delta E.

Demo

Consider this image:

Let us assume that your reference colors are a shade of green and a shade of magenta, whose RGB values are for example [0, 160, 0] and [120, 0, 140]. The following snippet identifies those image pixels whose Delta E with respect to the reference colors is lower than a certain threshold (15 and 20, respectively).

import numpy as np
from skimage import io
from skimage.color import rgb2lab, deltaE_cie76

rgb = io.imread('https://i.sstatic.net/npnrv.png')
lab = rgb2lab(rgb)

green = [0, 160, 0]
magenta = [120, 0, 140]

threshold_green = 15    
threshold_magenta = 20    

green_3d = np.uint8(np.asarray([[green]]))
magenta_3d = np.uint8(np.asarray([[magenta]]))

dE_green = deltaE_cie76(rgb2lab(green_3d), lab)
dE_magenta = deltaE_cie76(rgb2lab(magenta_3d), lab)

rgb[dE_green < threshold_green] = green_3d
rgb[dE_magenta < threshold_magenta] = magenta_3d
io.imshow(rgb)