pythonmachine-learningnlpzipf

Unable to Plot Zipf's Distribution Graph

I am new to python and machine learning. I want to plot Zipf's distribution graph for a text file. But my code gives error. Following is my python code

import re
from itertools import islice
#Get our corpus of medical words
frequency = {}
list(frequency)
open_file = open("abp.csv", 'r')
file_to_string = open_file.read()
words = re.findall(r'(\b[A-Za-z][a-z]{2,9}\b)', file_to_string)

#build dict of words based on frequency
for word in words:
    count = frequency.get(word,0)
    frequency[word] = count + 1


#limit words to 1000
n = 1000
frequency = {key:value for key,value in islice(frequency.items(), 0, n)}
#convert value of frequency to numpy array
s = frequency.values()
s = np.array(s)

#Calculate zipf and plot the data
a = 2. #  distribution parameter
count, bins, ignored = plt.hist(s[s<50], 50, normed=True)
x = np.arange(1., 50.)
y = x**(-a) / special.zetac(a)
plt.plot(x, y/max(y), linewidth=2, color='r')
plt.show()

And the above code gives the following error: count, bins, ignored = plt.hist(s[s<50], 50, normed=True)

TypeError: '<' not supported between instances of 'dict_values' and 'int'

Solution

  • The numpy array s actually consists of a dict_values object. To convert the values to a numpy array containing the numbers of the dict_values, use

    import numpy as np

frequency = {key:value for key,value in islice(frequency.items(), 0, n)}
s = np.fromiter(frequency.values(), dtype=float)

    Assuming, you want your array to consist of floats.

    For further information, read the docs.