I'm trying to deploy a CNN model created using Tensorflow to Heroku with FastAPI. The app runs on Heroku but returns an error when trying to make model predictions. Running heroku logs --tail returns this:
2022-02-17T03:32:12.426547+00:00 app[web.1]: [2022-02-17 03:32:12 +0000] [10] [ERROR] Exception in ASGI application
2022-02-17T03:32:12.426549+00:00 app[web.1]: Traceback (most recent call last):
2022-02-17T03:32:12.426549+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 373, in run_asgi
2022-02-17T03:32:12.426550+00:00 app[web.1]: result = await app(self.scope, self.receive, self.send)
2022-02-17T03:32:12.426550+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
2022-02-17T03:32:12.426551+00:00 app[web.1]: return await self.app(scope, receive, send)
2022-02-17T03:32:12.426551+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.9/site-packages/fastapi/applications.py", line 212, in __call__
2022-02-17T03:32:12.426552+00:00 app[web.1]: await super().__call__(scope, receive, send)
2022-02-17T03:32:12.426552+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.9/site-packages/starlette/applications.py", line 112, in __call__
2022-02-17T03:32:12.426552+00:00 app[web.1]: await self.middleware_stack(scope, receive, send)
2022-02-17T03:32:12.426553+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.9/site-packages/starlette/middleware/errors.py", line 181, in __call__
2022-02-17T03:32:12.426553+00:00 app[web.1]: raise exc
2022-02-17T03:32:12.426554+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.9/site-packages/starlette/middleware/errors.py", line 159, in __call__
2022-02-17T03:32:12.426554+00:00 app[web.1]: await self.app(scope, receive, _send)
2022-02-17T03:32:12.426554+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.9/site-packages/starlette/exceptions.py", line 82, in __call__
2022-02-17T03:32:12.426554+00:00 app[web.1]: raise exc
2022-02-17T03:32:12.426555+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.9/site-packages/starlette/exceptions.py", line 71, in __call__
2022-02-17T03:32:12.426555+00:00 app[web.1]: await self.app(scope, receive, sender)
2022-02-17T03:32:12.426555+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.9/site-packages/starlette/routing.py", line 656, in __call__
2022-02-17T03:32:12.426555+00:00 app[web.1]: await route.handle(scope, receive, send)
2022-02-17T03:32:12.426556+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.9/site-packages/starlette/routing.py", line 259, in handle
2022-02-17T03:32:12.426556+00:00 app[web.1]: await self.app(scope, receive, send)
2022-02-17T03:32:12.426556+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.9/site-packages/starlette/routing.py", line 61, in app
2022-02-17T03:32:12.426556+00:00 app[web.1]: response = await func(request)
2022-02-17T03:32:12.426557+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.9/site-packages/fastapi/routing.py", line 250, in app
2022-02-17T03:32:12.426557+00:00 app[web.1]: response = actual_response_class(response_data, **response_args)
2022-02-17T03:32:12.426557+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.9/site-packages/starlette/responses.py", line 49, in __init__
2022-02-17T03:32:12.426558+00:00 app[web.1]: self.body = self.render(content)
2022-02-17T03:32:12.426558+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.9/site-packages/starlette/responses.py", line 174, in render
2022-02-17T03:32:12.426558+00:00 app[web.1]: return json.dumps(
2022-02-17T03:32:12.426559+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.9/json/__init__.py", line 234, in dumps
2022-02-17T03:32:12.426559+00:00 app[web.1]: return cls(
2022-02-17T03:32:12.426559+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.9/json/encoder.py", line 199, in encode
2022-02-17T03:32:12.426560+00:00 app[web.1]: chunks = self.iterencode(o, _one_shot=True)
2022-02-17T03:32:12.426560+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.9/json/encoder.py", line 257, in iterencode
2022-02-17T03:32:12.426560+00:00 app[web.1]: return _iterencode(o, 0)
2022-02-17T03:32:12.426561+00:00 app[web.1]: ValueError: Out of range float values are not JSON compliant
Most notably on the last line, it says `ValueError: Out of range float values are not JSON compliant". My procfile looks like this:
web: gunicorn -w 2 -k uvicorn.workers.UvicornWorker main:app
When trying to run the server on WSL using python -m uvicorn main:app, I get this error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/home/eruaro/.local/lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 375, in run_asgi
result = await app(self.scope, self.receive, self.send)
File "/home/eruaro/.local/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
return await self.app(scope, receive, send)
File "/home/eruaro/.local/lib/python3.8/site-packages/fastapi/applications.py", line 212, in __call__
await super().__call__(scope, receive, send)
File "/home/eruaro/.local/lib/python3.8/site-packages/starlette/applications.py", line 112, in __call__
await self.middleware_stack(scope, receive, send)
File "/home/eruaro/.local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
raise exc
File "/home/eruaro/.local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
await self.app(scope, receive, _send)
File "/home/eruaro/.local/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
raise exc
File "/home/eruaro/.local/lib/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__
await self.app(scope, receive, sender)
File "/home/eruaro/.local/lib/python3.8/site-packages/starlette/routing.py", line 656, in __call__
await route.handle(scope, receive, send)
File "/home/eruaro/.local/lib/python3.8/site-packages/starlette/routing.py", line 259, in handle
await self.app(scope, receive, send)
File "/home/eruaro/.local/lib/python3.8/site-packages/starlette/routing.py", line 61, in app
response = await func(request)
File "/home/eruaro/.local/lib/python3.8/site-packages/fastapi/routing.py", line 250, in app
response = actual_response_class(response_data, **response_args)
File "/home/eruaro/.local/lib/python3.8/site-packages/starlette/responses.py", line 49, in __init__
self.body = self.render(content)
File "/home/eruaro/.local/lib/python3.8/site-packages/starlette/responses.py", line 174, in render
return json.dumps(
File "/usr/lib/python3.8/json/__init__.py", line 234, in dumps
return cls(
File "/usr/lib/python3.8/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
ValueError: Out of range float values are not JSON compliant
It's similar to the one I get on Heroku. However, on Windows, using the same command like the one I used on WSL, the app works. No error is returned and I'm able to make predictions on the server. How do I remove the JSON compliant error?
For reference, my code is in a singular main.py file:
from fastapi import FastAPI
from tensorflow.keras.models import load_model
from tensorflow.keras.utils import get_file
from tensorflow.keras.utils import load_img
from tensorflow.keras.utils import img_to_array
from tensorflow import expand_dims
from tensorflow.nn import softmax
from numpy import argmax
from numpy import max
from numpy import array
app = FastAPI()
model_dir = "food-vision-model.h5"
model = load_model(model_dir)
class_predictions = array([
'apple_pie',
'baby_back_ribs',
'baklava',
'beef_carpaccio',
'beef_tartare',
'beet_salad',
'beignets',
'bibimbap',
'bread_pudding',
'breakfast_burrito',
'bruschetta',
'caesar_salad',
'cannoli',
'caprese_salad',
'carrot_cake',
'ceviche',
'cheesecake',
'cheese_plate',
'chicken_curry',
'chicken_quesadilla',
'chicken_wings',
'chocolate_cake',
'chocolate_mousse',
'churros',
'clam_chowder',
'club_sandwich',
'crab_cakes',
'creme_brulee',
'croque_madame',
'cup_cakes',
'deviled_eggs',
'donuts',
'dumplings',
'edamame',
'eggs_benedict',
'escargots',
'falafel',
'filet_mignon',
'fish_and_chips',
'foie_gras',
'french_fries',
'french_onion_soup',
'french_toast',
'fried_calamari',
'fried_rice',
'frozen_yogurt',
'garlic_bread',
'gnocchi',
'greek_salad',
'grilled_cheese_sandwich',
'grilled_salmon',
'guacamole',
'gyoza',
'hamburger',
'hot_and_sour_soup',
'hot_dog',
'huevos_rancheros',
'hummus',
'ice_cream',
'lasagna',
'lobster_bisque',
'lobster_roll_sandwich',
'macaroni_and_cheese',
'macarons',
'miso_soup',
'mussels',
'nachos',
'omelette',
'onion_rings',
'oysters',
'pad_thai',
'paella',
'pancakes',
'panna_cotta',
'peking_duck',
'pho',
'pizza',
'pork_chop',
'poutine',
'prime_rib',
'pulled_pork_sandwich',
'ramen',
'ravioli',
'red_velvet_cake',
'risotto',
'samosa',
'sashimi',
'scallops',
'seaweed_salad',
'shrimp_and_grits',
'spaghetti_bolognese',
'spaghetti_carbonara',
'spring_rolls',
'steak',
'strawberry_shortcake',
'sushi',
'tacos',
'takoyaki',
'tiramisu',
'tuna_tartare',
'waffles'
])
@app.get("/")
async def root():
return {"message": "Welcome to the Food Vision API!"}
@app.post("/net/image/prediction/")
async def get_net_image_prediction(image_link: str = ""):
if image_link == "":
return {"message": "No image link provided"}
img_path = get_file(
origin = image_link
)
img = load_img(
img_path,
target_size = (224, 224)
)
img_array = img_to_array(img)
img_array = expand_dims(img_array, 0)
pred = model.predict(img_array)
score = softmax(pred[0])
class_prediction = class_predictions[argmax(score)]
model_score = round(max(score) * 100, 2)
return {
"model_prediction_class": class_prediction,
"model_prediction_score": model_score
}
I've solved the issue, the solution is to dockerize the whole application and then deploy it to Heroku. That way it can work when running on WSL (Linux), and by extension on Heroku as well (which uses Linux).
This thus requires editing the main.py file by a little bit:
from fastapi import FastAPI
from tensorflow.keras.models import load_model
from tensorflow.keras.utils import get_file
from tensorflow.keras.utils import load_img
from tensorflow.keras.utils import img_to_array
from tensorflow import expand_dims
from tensorflow.nn import softmax
from numpy import argmax
from numpy import max
from numpy import array
from json import dumps
from uvicorn import run
import os
app = FastAPI()
model_dir = "food-vision-model.h5"
model = load_model(model_dir)
class_predictions = array([
'apple_pie',
'baby_back_ribs',
'baklava',
'beef_carpaccio',
'beef_tartare',
'beet_salad',
'beignets',
'bibimbap',
'bread_pudding',
'breakfast_burrito',
'bruschetta',
'caesar_salad',
'cannoli',
'caprese_salad',
'carrot_cake',
'ceviche',
'cheesecake',
'cheese_plate',
'chicken_curry',
'chicken_quesadilla',
'chicken_wings',
'chocolate_cake',
'chocolate_mousse',
'churros',
'clam_chowder',
'club_sandwich',
'crab_cakes',
'creme_brulee',
'croque_madame',
'cup_cakes',
'deviled_eggs',
'donuts',
'dumplings',
'edamame',
'eggs_benedict',
'escargots',
'falafel',
'filet_mignon',
'fish_and_chips',
'foie_gras',
'french_fries',
'french_onion_soup',
'french_toast',
'fried_calamari',
'fried_rice',
'frozen_yogurt',
'garlic_bread',
'gnocchi',
'greek_salad',
'grilled_cheese_sandwich',
'grilled_salmon',
'guacamole',
'gyoza',
'hamburger',
'hot_and_sour_soup',
'hot_dog',
'huevos_rancheros',
'hummus',
'ice_cream',
'lasagna',
'lobster_bisque',
'lobster_roll_sandwich',
'macaroni_and_cheese',
'macarons',
'miso_soup',
'mussels',
'nachos',
'omelette',
'onion_rings',
'oysters',
'pad_thai',
'paella',
'pancakes',
'panna_cotta',
'peking_duck',
'pho',
'pizza',
'pork_chop',
'poutine',
'prime_rib',
'pulled_pork_sandwich',
'ramen',
'ravioli',
'red_velvet_cake',
'risotto',
'samosa',
'sashimi',
'scallops',
'seaweed_salad',
'shrimp_and_grits',
'spaghetti_bolognese',
'spaghetti_carbonara',
'spring_rolls',
'steak',
'strawberry_shortcake',
'sushi',
'tacos',
'takoyaki',
'tiramisu',
'tuna_tartare',
'waffles'
])
@app.get("/")
async def root():
return {"message": "Welcome to the Food Vision API!"}
@app.post("/net/image/prediction/")
async def get_net_image_prediction(image_link: str = ""):
if image_link == "":
return {"message": "No image link provided"}
img_path = get_file(
origin = image_link
)
img = load_img(
img_path,
target_size = (224, 224)
)
img_array = img_to_array(img)
img_array = expand_dims(img_array, 0)
pred = model.predict(img_array)
score = softmax(pred[0])
class_prediction = class_predictions[argmax(score)]
model_score = round(max(score) * 100, 2)
model_score = dumps(model_score.tolist())
return {
"model_prediction_class": class_prediction,
"model_prediction_score": model_score
}
if __name__ == "__main__":
port = int(os.environ.get('PORT', 5000))
run(app, host="0.0.0.0", port=port)
The main difference here is the use of the os.environ.get(). I found that just manually defining a port like port = 5000 returns an R10 (boot timeout) error on Heroku.
From here, I created a Dockerfile with the following contents:
FROM python:3.7.3-stretch
# Maintainer info
LABEL maintainer="erickson_ruaroii@dlsu.edu.ph"
# Make working directories
RUN mkdir -p /my-directory/project
WORKDIR /my-directory/project
# Upgrade pip with no cache
RUN pip install --no-cache-dir -U pip
# Copy application requirements file to the created working directory
COPY requirements.txt .
# Install application dependencies from the requirements file
RUN pip install -r requirements.txt
# Copy every file in the source folder to the created working directory
COPY . .
# Run the python application
CMD ["python", "main.py"]
With requirements.txt having these:
fastapi==0.73.0
gunicorn==20.1.0
numpy==1.19.5
uvicorn==0.15.0
image==1.5.33
tensorflow-cpu==2.7.0
I used tensorflow-cpu to bypass the slug size and memory limit of Heroku on a free account.
From here I was then able to build the docker image and deploy it to heroku.
$ docker image build -t app-name .
$ heroku create app-name
$ heroku container:push web --app app-name
$ heroku container:release web --app food-vision-api
When running locally, I used the command below:
$ docker run -p 5000:5000 -d app-name