I was trying to make an Image Captioning model in a similar fashion as in here
I used ResNet50 instead off VGG16 and also had to use progressive loading via model.fit_generator() method.
I used ResNet50 from here and when I imported it by setting include_top = False, It gave me features of photo in shape of {'key': [[[[value1, value2, .... value 2048]]]]}, where "key" is the image id.
Here's my code of caption generator function:-
def createCaptions(tokenizer, photoData, MaxLength, model):
for key, feature in photoData.items():
inSeq = "START"
for i in range(MaxLength):
sequence = tokenizer.texts_to_sequences([inSeq])[0]
sequence = pad_sequences([sequence], maxlen = MaxLength)
ID = model.predict([np.array(feature[0][0][0]), inSeq])
ID = word_for_id(ID)
if ID is None:
break
inSeq += " " + ID
if ID == "END":
break
print(inSeq)
The function word_for_id is :-
def word_for_id(integer, tokenizer):
for word, index in tokenizer.word_index.items():
if index == integer:
return word
return None
I had generated photoData via:-
features = {}
for images in os.listdir(args["image"]):
filename = args["image"] + '/' + images
image = load_img(filename, target_size = inputShape)
image = img_to_array(image)
image = np.expand_dims(image, axis = 0)
image = preprocess(image)
pred = resnet.predict(image)
image_id = images.split('.')[0]
features[image_id] = pred
print('>{}'.format(images))
features is my photoData dictionary.
When I try to generate captions:-
caption = createCaptions(tokenizerTrain, features, 34, model)
I get the following error:-
Traceback (most recent call last):
File "CaptionGenerator.py", line 111, in <module>
caption = createCaptions(tokenizerTrain, features, 34, model)
File "CaptionGenerator.py", line 101, in createCaptions
ID = model.predict([np.array(feature[0][0][0]), inSeq])
File "/home/aditya/.virtualenvs/cv/lib/python3.5/site-packages/keras/engine/training.py", line 1817, in predict
check_batch_axis=False)
File "/home/aditya/.virtualenvs/cv/lib/python3.5/site-packages/keras/engine/training.py", line 76, in _standardize_input_data
data = [np.expand_dims(x, 1) if x is not None and x.ndim == 1 else x for x in data]
File "/home/aditya/.virtualenvs/cv/lib/python3.5/site-packages/keras/engine/training.py", line 76, in <listcomp>
data = [np.expand_dims(x, 1) if x is not None and x.ndim == 1 else x for x in data]
AttributeError: 'str' object has no attribute 'ndim'
Where did I go wrong? Please help. Thanks in advance.
You pass inSeq = "START"
to model.predict
as a string:
ID = model.predict([np.array(feature[0][0][0]), inSeq])
without pre-processing it. You will need to encode it to an array.