from sklearn.feature_extraction.text import CountVectorizer
import pandas as pd
headers = ['label', 'sms_message']
df = pd.read_csv ('spam.csv', names = headers)
df ['label'] = df['label'].map({'ham': 0, 'spam': 1})
print (df.head(7))
print (df.shape)
count_vector = CountVectorizer()
#count_vector.fit(df)
y = count_vector.fit_transform(df)
count_vector.get_feature_names()
doc_array = y.toarray()
print (doc_array)
frequency_matrix = pd.DataFrame(doc_array, columns = count_vector.get_feature_names())
frequency_matrix
Sample data and output:
label sms_message
0 0 Go until jurong point, crazy.. Available only ...
1 0 Ok lar... Joking wif u oni...
2 1 Free entry in 2 a wkly comp to win FA Cup fina...
3 0 U dun say so early hor... U c already then say...
(5573, 2)
[[1 0]
[0 1]]
label sms_message
0 1 0
1 0 1
My Question:
My csv file is basically many rows of sms messages.
I cannot understand why I am getting only output for the column labels and not for the entire rows of sms texts.
Thank you for any help.
Pass only the sms_message column to count vectorizer as shown below.
import numpy as np
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
docs = ['Tea is an aromatic beverage..',
'After water, it is the most widely consumed drink in the world',
'There are many different types of tea.',
'Tea has a stimulating effect in humans.',
'Tea originated in Southwest China during the Shang dynasty']
df = pd.DataFrame({'sms_message': docs, 'label': np.random.choice([0, 1], size=5)})
cv = CountVectorizer()
counts = cv.fit_transform(df['sms_message'])
df_counts = pd.DataFrame(counts.toarray(), columns=cv.get_feature_names_out())
df_counts['label'] = df['label']
Output:
df_counts
Out[26]:
after an are aromatic beverage ... types water widely world label
0 0 1 0 1 1 ... 0 0 0 0 1
1 1 0 0 0 0 ... 0 1 1 1 0
2 0 0 1 0 0 ... 1 0 0 0 1
3 0 0 0 0 0 ... 0 0 0 0 1
4 0 0 0 0 0 ... 0 0 0 0 0
[5 rows x 32 columns]