I am working on a sentiment analysis topic and there are a lot of comments with emojis.
I would like to know if my code is correct or is there a way to optimize it as well?
Code to do smiley count
import pandas as pd
import regex as re
import emoji
# Assuming your DataFrame is called 'df' and the column with comments is 'Document'
comments = df['Document']
# Initialize an empty dictionary to store smiley counts and types
smiley_data = {'Smiley': [], 'Count': [], 'Type': []}
# Define a regular expression pattern to match smileys
pattern = r'([\U0001F600-\U0001F64F\U0001F300-\U0001F5FF\U0001F680-\U0001F6FF\U0001F1E0-\U0001F1FF])'
# Iterate over the comments
for comment in comments:
# Extract smileys and their types from the comment
smileys = re.findall(pattern, comment)
# Increment the count and store the smileys and their types
for smiley in smileys:
if smiley in smiley_data['Smiley']:
index = smiley_data['Smiley'].index(smiley)
smiley_data['Count'][index] += 1
else:
smiley_data['Smiley'].append(smiley)
smiley_data['Count'].append(1)
smiley_data['Type'].append(emoji.demojize(smiley))
# Create a DataFrame from the smiley data
smiley_df = pd.DataFrame(smiley_data)
# Sort the DataFrame by count in descending order
smiley_df = smiley_df.sort_values(by='Count', ascending=False)
# Print the smiley data
smiley_df
I am majorly not sure if my below code block is getting all the smileys
# Define a regular expression pattern to match smileys
pattern = r'([\U0001F600-\U0001F64F\U0001F300-\U0001F5FF\U0001F680-\U0001F6FF\U0001F1E0-\U0001F1FF])'
would like to know what can I do with this analysis. something else on top of it - some charts maybe?
I am also sharing a test dataset that will generate similar smiley counts as those available in my real data. Please note that the test dataset only has known smileys if there is something else. it won't be there like in a real dataset.
Test Dataset
import random
import pandas as pd
smileys = ['ð', 'ð', 'ð', 'ðŧ', 'ð', 'ð', 'ð', 'ð', 'ðž', 'ðĐ']
# Additional smileys to complete the required count
additional_smileys = ['ð', 'ð', 'ðĪĐ', 'ð', 'ðĪ', 'ð', 'ð', 'ð', 'ð', 'ðĨģ', 'ð', 'ð', 'ðĨ', 'ðĨ°', 'ðĪŠ', 'ð', 'ðĪ',
'ð', 'ðĪ', 'ðĪŦ', 'ð', 'ðĨą', 'ðĨķ', 'ðĪŪ', 'ðĪĄ', 'ð', 'ðī', 'ð', 'ðŪ', 'ðĪĨ', 'ðĒ', 'ðĪ', 'ð', 'ð',
'ð―', 'ðĪ', 'ðĶ', 'ðž', 'ðĩ', 'ðĶ', 'ðļ', 'ðĶ']
# Combine the required smileys and additional smileys
all_smileys = smileys + additional_smileys
# Set a random seed for reproducibility
random.seed(42)
# Generate a single review
def generate_review(with_smiley=False):
review = "This movie"
if with_smiley:
review += " " + random.choice(all_smileys)
review += " is "
review += random.choice(["amazing", "excellent", "fantastic", "brilliant", "great", "good", "okay", "average",
"mediocre", "disappointing", "terrible", "awful", "horrible"])
review += random.choice(["!", "!!", "!!!", ".", "..", "..."]) + " "
review += random.choice(["Highly recommended", "Definitely worth watching", "A must-see", "I loved it",
"Not worth your time", "Skip it"]) + random.choice(["!", "!!", "!!!"])
return review
# Generate the random dataset
def generate_dataset():
dataset = []
review_count = 5000
# Generate reviews with top smileys
for smiley, count, _ in top_smileys:
while count > 0:
review = generate_review(with_smiley=True)
if smiley in review:
dataset.append(review)
count -= 1
# Generate reviews with additional smileys
additional_smileys_count = len(additional_smileys)
additional_smileys_per_review = review_count - len(dataset)
additional_smileys_per_review = min(additional_smileys_per_review, additional_smileys_count)
for _ in range(additional_smileys_per_review):
review = generate_review(with_smiley=True)
dataset.append(review)
# Generate reviews without smileys
while len(dataset) < review_count:
review = generate_review()
dataset.append(review)
# Shuffle the dataset
random.shuffle(dataset)
return dataset
# List of top smileys and their counts
top_smileys = [
('ð', 331, ':thumbs_up:'),
('ð', 50, ':OK_hand:'),
('ð', 41, ':smiling_face_with_heart-eyes:'),
('ðŧ', 38, ':light_skin_tone:'),
('ð', 35, ':smiling_face_with_smiling_eyes:'),
('ð', 14, ':slightly_smiling_face:'),
('ð', 12, ':thumbs_down:'),
('ð', 12, ':grinning_face_with_big_eyes:'),
('ðž', 10, ':medium-light_skin_tone:'),
('ðĐ', 10, ':pile_of_poo:')
]
# Generate the dataset
dataset = generate_dataset()
# Create a data frame with 'Document' column
df = pd.DataFrame({'Document': dataset})
# Display the DataFrame
df
Thank you in advance!
Update
If you prefer to use emoji package, you can do:
import emoji
text = df['Document'].str.cat(sep='\n')
out = (pd.DataFrame(emoji.emoji_list(text)).value_counts('emoji')
.rename_axis('Smiley').rename('Count').reset_index()
.assign(Type=lambda x: x['Smiley'].apply(emoji.demojize)))
Output:
>>> out
Smiley Count Type
0 ð 331 :thumbs_up:
1 ð 50 :OK_hand:
2 ðŧ 41 :light_skin_tone:
3 ð 41 :smiling_face_with_heart-eyes:
4 ð 35 :smiling_face_with_smiling_eyes:
5 ð 15 :slightly_smiling_face:
6 ð 14 :thumbs_down:
7 ð 13 :grinning_face_with_big_eyes:
8 ðž 10 :medium-light_skin_tone:
9 ðĐ 10 :pile_of_poo:
10 ð 3 :winking_face_with_tongue:
11 ðĶ 3 :owl:
12 ðĪ 2 :robot:
13 ð 2 :expressionless_face:
14 ð― 2 :alien:
15 ðĪŦ 2 :shushing_face:
16 ðĒ 2 :crying_face:
17 ðĪŠ 2 :zany_face:
18 ð 2 :see-no-evil_monkey:
19 ð 2 :speak-no-evil_monkey:
20 ð 1 :smiling_face_with_halo:
21 ðĪŪ 1 :face_vomiting:
22 ðĪ 1 :face_with_hand_over_mouth:
23 ðĪĄ 1 :clown_face:
24 ðĪ 1 :smiling_face_with_open_hands:
25 ð 1 :face_with_rolling_eyes:
26 ð 1 :grinning_squinting_face:
27 ðļ 1 :frog:
28 ðŪ 1 :face_with_open_mouth:
29 ðž 1 :panda:
30 ð 1 :kissing_face_with_closed_eyes:
31 ð 1 :smiling_face_with_sunglasses:
32 ð 1 :face_blowing_a_kiss:
You can use str.extractall to avoid a loop then use value_counts to count the number of occurences. Finally, "demojize" each smiley (the slowest part):
out = (df['Document'].str.extractall(pattern).value_counts()
.rename_axis('Smiley').rename('Count').reset_index()
.assign(Type=lambda x: x['Smiley'].apply(emoji.demojize)))
Output:
>>> out
Smiley Count Type
0 ð 331 :thumbs_up:
1 ð 50 :OK_hand:
2 ðŧ 41 :light_skin_tone:
3 ð 41 :smiling_face_with_heart-eyes:
4 ð 35 :smiling_face_with_smiling_eyes:
5 ð 15 :slightly_smiling_face:
6 ð 14 :thumbs_down:
7 ð 13 :grinning_face_with_big_eyes:
8 ðĐ 10 :pile_of_poo:
9 ðž 10 :medium-light_skin_tone:
10 ð 3 :winking_face_with_tongue:
11 ð 2 :expressionless_face:
12 ð 2 :see-no-evil_monkey:
13 ðĒ 2 :crying_face:
14 ð 2 :speak-no-evil_monkey:
15 ð― 2 :alien:
16 ð 1 :smiling_face_with_sunglasses:
17 ð 1 :face_blowing_a_kiss:
18 ð 1 :kissing_face_with_closed_eyes:
19 ðļ 1 :frog:
20 ð 1 :smiling_face_with_halo:
21 ðŪ 1 :face_with_open_mouth:
22 ð 1 :grinning_squinting_face:
23 ð 1 :face_with_rolling_eyes:
24 ðž 1 :panda:
The pattern part is correct? I am not missing out on any emoticons?
Your pattern is not right. I don't know the full list you want to extract but below you have a code to debug it:
# add latin1 codes --v
pattern2 = '([\\U00000000-\\U000000FF\\U0001F600-\\U0001F64F\\U0001F300-\\U0001F5FF\\U0001F680-\\U0001F6FF\\U0001F1E0-\\U0001F1FF])'
other = df['Document'].str.replace(pattern2, '', regex=True)
print(other[other != ''])
# Output / Missed emojis
1149 ðĪ
1238 ðĶ
1305 ðĪŦ
1424 ðĪŦ
1978 ðĪ
2611 ðĪŪ
2623 ðĶ
2959 ðĪĄ
3717 ðĪŠ
4045 ðĶ
4067 ðĪ
4699 ðĪ
4975 ðĪŠ
Name: Document, dtype: object