pythonpandasscikit-learnmulti-indexrecord-linkage

How to implement a SVM model when the predicted values are pairs of indexes that match?


I am trying to fit a SVM model where my predicted true values are multiindexes that match. The problem is that I dont know how to specify that the multiindixes are the true values.

I cannot use the record linkage classification step because it is not very flexible.

from sklearn.svm import SVC

golden_pairs = filter_tests_new_df[:training_value]
golden_matches_index = golden_pairs[golden_pairs['ev_2'] == 1].index 
# This is a multiindex type

svm = SVC(gamma='auto')
svm.fit(golden_pairs, golden_matches_index) 
# I dont know how to specify that the golden_matches_index are the good matches

# Predict the match status for all record pairs
result_svm = svm.predict(test_pairs[columns_to_keep])

Solution

  • You don't have to specify the index, instead use the generated boolean Series as the labels for the classification.

    Here's an example.

    # Sample data
    data = pd.DataFrame({'a': [1, 2, 3], 
                         'b': [1, 1, 0]})
    
    data
       a  b
    0  1  1
    1  2  1
    2  3  0
    
    # Generating labels
    data['b'] == 1
    0     True
    1     True
    2    False
    Name: b, dtype: bool
    
    # Can convert them to integer if required
    (data['b'] == 1).astype(int)
    0    1
    1    1
    2    0
    Name: b, dtype: int64
    

    Based on your code, I think this should do the trick

    # Boolean
    golden_pairs['ev_2'] == 1
    
    # Integer
    (golden_pairs['ev_2'] == 1).astype(int)