I am trying to apply Lemmatization after I tokenized my "script" column. But I get an AttributeError. I tried different thins
Here is my "script" column:
df_toklem["script"][0:5]
---------------------------------------------------------------------------
type(df_toklem["script"])
Output:
id
1 [ext, street, day, ups, man, big, pot, belly, ...
2 [credits, still, life, tableaus, lawford, n, h...
3 [fade, ext, convent, day, whispering, nuns, pr...
4 [fade, int, c, hercules, turbo, prop, night, e...
5 [open, theme, jaws, plane, busts, clouds, like...
Name: script, dtype: object
---------------------------------------------------------------------------
pandas.core.series.Series
And the code where I try to apply Lemmatization:
from textblob import Word
nltk.download("wordnet")
df_toklem["script"].apply(lambda x: " ".join([Word(word).lemmatize() for word in x.split()]))
ERROR:
[nltk_data] Downloading package wordnet to
[nltk_data] C:\Users\PC\AppData\Roaming\nltk_data...
[nltk_data] Package wordnet is already up-to-date!
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-72-dbc80c619ec5> in <module>
1 from textblob import Word
2 nltk.download("wordnet")
----> 3 df_toklem["script"].apply(lambda x: " ".join([Word(word).lemmatize() for word in x.split()]))
~\Anaconda3\lib\site-packages\pandas\core\series.py in apply(self, func, convert_dtype, args, **kwds)
4198 else:
4199 values = self.astype(object)._values
-> 4200 mapped = lib.map_infer(values, f, convert=convert_dtype)
4201
4202 if len(mapped) and isinstance(mapped[0], Series):
pandas\_libs\lib.pyx in pandas._libs.lib.map_infer()
<ipython-input-72-dbc80c619ec5> in <lambda>(x)
1 from textblob import Word
2 nltk.download("wordnet")
----> 3 df_toklem["script"].apply(lambda x: " ".join([Word(word).lemmatize() for word in x.split()]))
AttributeError: 'WordList' object has no attribute 'split'
I tried different things but unfortunately couldn't find an efficient solution. Thank you for your time.
What you are trying to do won't work because you are applying a string function (split) to a Word List.
I would try to use nltk
, instead, and create a new pandas column with my tokenized data:
import nltk
df_toklem['tokenized'] = df_toklem.apply(lambda row: nltk.word_tokenize(row['script']))