I have some SAS coding that I am trying to convert to Python. I am having difficulties calculating the jaccard distance on asymmetric data – where the zeros should be ignored in the calculation. I do find some examples on jaccard but they do not calculate the asymmetric distance. Just checking to see if a library has this available before I try to reinvent the wheel. If someone could please steer me in the right direction, I would really appreciate it.
My test dataset contains 5 headers and 5 rows
H0 H1 H2 H3 H4
A 1 1 1 1 0
B 1 0 1 1 0
C 1 1 1 1 0
D 0 0 1 1 1
E 1 1 0 1 0
below is the expected result(distance) calculated by shorthand and also from using SAS:
. | A | B | C | D | E
A | 0 | 0.25| 0 | 0.6 | 0.25
B | 0.25| 0 | 0.25| 0.5 | 0.5
C | 0 | 0.25| 0 | 0.6 | 0.25
D | 0.6 | 0.5 | 0.6 | 0 | 0.8
E | 0.25| 0.5 | 0.25| 0.8 | 0
But, using jaccard in python, I get results like:
. |A | B | C | D | E
A |1.00 | 0.43 | 0.61 | 0.55 | 0.46
B |0.43 | 1.00 | 0.52 | 0.56 | 0.49
C |0.61 | 0.52 | 1.00 | 0.48 | 0.53
D |0.55 | 0.56 | 0.48 | 1.00 | 0.49
E |0.46 | 0.49 | 0.53 | 0.49 | 1.00
Below is the code I experimented on. I am new to Python so I might be making an obvious mistake. I have added the SAS code at the bottom in case someone would like it for reference:
Python Code:
np.random.seed(0)
df = pd.DataFrame(np.random.binomial(1, 0.5, size=(100, 5)),
columns=list('ABCDE'))
print(df.head())
jac_sim = 1 - pairwise_distances(df.T, metric = "jaccard")
jac_sim = pd.DataFrame(jac_sim, index=df.columns, columns=df.columns)
import itertools
sim_df = pd.DataFrame(np.ones((5, 5)), index=df.columns, columns=df.columns)
for col_pair in itertools.combinations(df.columns, 2):
sim_df.loc[col_pair] = sim_df.loc[tuple(reversed(col_pair))] =
jaccard_similarity_score(df[col_pair[0]], df[col_pair[1]])
print(sim_df)
SAS Code:
proc import datafile = '/home/xxx/xxx.csv'
out = work.Binary2 replace
dbms = CSV;
GUESSINGROWS=MAX;
run;
proc sort;
by VAR1;
run;
title ’Data Clustering of BN’;
proc distance data=Binary2 method=djaccard absent=0 out=distjacc;
var anominal (r0--r4);
id VAR1;
run;
I found some obvious mistakes. First thing is that you need to create matrix of size=(5,5):
import pandas as pd
import numpy as np
from sklearn.metrics import pairwise_distances, jaccard_similarity_score
np.random.seed(0)
df = pd.DataFrame(np.random.binomial(1, 0.5, size=(5, 5)).T, columns=list('ABCDE'))
print(df.T)
Second thing is that if you print just head, you do not see that matrix has more than 5 rows. With just 5 lines, these two:
print(df.T.head())
print(df.T)
print the same result:
0 1 2 3 4
A 1 1 1 1 0
B 1 0 1 1 0
C 1 1 1 1 0
D 0 0 1 1 1
E 1 1 0 1 0
After the above change it is possible to use pairwise_distances:
jac_sim = pairwise_distances(df.T.astype(bool), metric = "jaccard")
jac_sim = pd.DataFrame(jac_sim, index=df.columns, columns=df.columns)
print(jac_sim)
in order to obtain the desired result:
A B C D E
A 0.00 0.25 0.00 0.6 0.25
B 0.25 0.00 0.25 0.5 0.50
C 0.00 0.25 0.00 0.6 0.25
D 0.60 0.50 0.60 0.0 0.80
E 0.25 0.50 0.25 0.8 0.00
There is also .astype(bool) in the above code in order to prevent warning when running pairwise_distance.
It is necessary to be careful in applying transposes .T, as pairwise_distance seem to work rather with columns than with rows.
With function jaccard_similarity_score
import itertools
sim_df = pd.DataFrame(np.zeros((5, 5)), index=df.columns, columns=df.columns)
for col_pair in itertools.combinations(df.columns, 2):
sim_df.loc[col_pair] = sim_df.loc[tuple(reversed(col_pair))] = \
1 - jaccard_similarity_score(df[col_pair[0]], df[col_pair[1]], normalize = True)
print(sim_df)
I got a different matrix:
A B C D E
A 0.0 0.2 0.0 0.6 0.2
B 0.2 0.0 0.2 0.4 0.4
C 0.0 0.2 0.0 0.6 0.2
D 0.6 0.4 0.6 0.0 0.8
E 0.2 0.4 0.2 0.8 0.0
Looking more closely jaccard_similarity_score:
print(df['A'])
print(df['B'])
jaccard_similarity_score(df['A'], df['B'], normalize = True)
reveals that zeros were not excluded the result:
0 1
1 1
2 1
3 1
4 0
Name: A, dtype: int32
0 1
1 0
2 1
3 1
4 0
Name: B, dtype: int32
Out[123]: 0.8
Because the result is 4 similar / 5 total = 0.8, not 3 similar nonzeros / 4 total nonzeros = 0.75.