I'm looking for a way to split a text into n-grams. Normally I would do something like:
import nltk
from nltk import bigrams
string = "I really like python, it's pretty awesome."
string_bigrams = bigrams(string)
print string_bigrams
I am aware that nltk only offers bigrams and trigrams, but is there a way to split my text in four-grams, five-grams or even hundred-grams?
Thanks!
Great native python based answers given by other users. But here's the nltk
approach (just in case, the OP gets penalized for reinventing what's already existing in the nltk
library).
There is an ngram module that people seldom use in nltk
. It's not because it's hard to read ngrams, but training a model base on ngrams where n > 3
will result in much data sparsity.
from nltk import ngrams
sentence = 'this is a foo bar sentences and I want to ngramize it'
n = 6
sixgrams = ngrams(sentence.split(), n)
for grams in sixgrams:
print(grams)