pythonscikit-learn

Implementing Bag of Words in scikit-learn


from sklearn.feature_extraction.text import CountVectorizer
import pandas as pd
headers = ['label', 'sms_message']
df = pd.read_csv ('spam.csv', names = headers)
df ['label'] = df['label'].map({'ham': 0, 'spam': 1})
print (df.head(7))
print (df.shape)
count_vector = CountVectorizer()
#count_vector.fit(df)
y = count_vector.fit_transform(df)
count_vector.get_feature_names()
doc_array = y.toarray()
print (doc_array)
frequency_matrix = pd.DataFrame(doc_array, columns = count_vector.get_feature_names())
frequency_matrix

Sample data and output:

   label                                        sms_message
0      0  Go until jurong point, crazy.. Available only ...
1      0                      Ok lar... Joking wif u oni...
2      1  Free entry in 2 a wkly comp to win FA Cup fina...
3      0  U dun say so early hor... U c already then say...

(5573, 2)
[[1 0]
 [0 1]]

label   sms_message
0   1   0
1   0   1

My Question:

My csv file is basically many rows of sms messages.

I cannot understand why I am getting only output for the column labels and not for the entire rows of sms texts.

Thank you for any help.


Solution

  • Pass only the sms_message column to count vectorizer as shown below.

    import numpy as np
    import pandas as pd
    from sklearn.feature_extraction.text import CountVectorizer
    
    docs = ['Tea is an aromatic beverage..',
            'After water, it is the most widely consumed drink in the world',
            'There are many different types of tea.',
            'Tea has a stimulating effect in humans.',
            'Tea originated in Southwest China during the Shang dynasty'] 
    
    df = pd.DataFrame({'sms_message': docs, 'label': np.random.choice([0, 1], size=5)})
    
    cv = CountVectorizer()
    counts = cv.fit_transform(df['sms_message'])
    
    df_counts = pd.DataFrame(counts.toarray(), columns=cv.get_feature_names_out())
    df_counts['label'] = df['label']
    

    Output:

    df_counts
    
    Out[26]: 
       after  an  are  aromatic  beverage  ...  types  water  widely  world  label
    0      0   1    0         1         1  ...      0      0       0      0      1
    1      1   0    0         0         0  ...      0      1       1      1      0
    2      0   0    1         0         0  ...      1      0       0      0      1
    3      0   0    0         0         0  ...      0      0       0      0      1
    4      0   0    0         0         0  ...      0      0       0      0      0
    
    [5 rows x 32 columns]