I want to access two services with my Kubernetes ingress, and both services are based upon a Flask app. Each Flask app is made of a main.py script and a index.html web page, which is rendered using render_template library. Each service should let me get an answer from a Large Linguage Model (LLM) using Groq, and each service differs from the other only because of the model it uses, so I will show you the code of just one of them.
main.py
from flask import Flask, render_template, request
from groq import Groq
model_id="qwen/qwen3-32b"
groq_api_key = "<--my API key :)-->"
# Initialize the Groq client with the API key
client = Groq(api_key=groq_api_key)
app = Flask(__name__)
# Home route to display
@app.route('/')
def index():
return render_template('index.html')
# Route to handle form submission
@app.route('/answer', methods=['POST'])
def answer():
input_text = request.form.get('input_text')
if not input_text:
return "Please provide input text.", 400
try:
completion = client.chat.completions.create(
model=model_id,
messages=[
{"role": "system", "content": "User chatbot"},
{"role": "user", "content": input_text}
],
temperature=1,
max_tokens=1024,
top_p=1,
stream=True,
stop=None,
)
# Collect the streamed response
result = ""
for chunk in completion:
result += chunk.choices[0].delta.content or ""
except Exception as e:
return f"An error occurred: {e}", 500
# Render the index.html template with the results
return render_template('index.html', input_text=input_text, result=result)
if __name__ == "__main__":
app.run(host="0.0.0.0", port=5000)
index.html
<form action="{{ url_for('answer') }}" method="POST">
<label for="input_text">Enter Input Text:</label>
<br>
<textarea id="input_text" name="input_text" rows="4" cols="50" required></textarea>
<button type="submit">Submit</button>
</form>
{% if result %}
<div class="result-container">
<p><strong>Input Text:</strong> {{ input_text }}</p>
<p><strong>Result:</strong> {{ result }}</p>
</div>
{% endif %}
Similarly, for each Flask app there is a Deployment and a Service. The Deployment uses a simple Docker image with the Flask app and its dependencies inside. Since the two Deployments and the two Services are similar, I will show you the code of just one of them (each).
Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: groq-app
spec:
selector:
matchLabels:
app: groq-app
template:
metadata:
labels:
app: groq-app
spec:
containers:
- name: groq-app
image: <--my DockerHub username :)-->/groq-test:v2
ports:
- containerPort: 5000
Service:
apiVersion: v1
kind: Service
metadata:
name: groq-app-service
spec:
type: NodePort
selector:
app: groq-app
ports:
- name: http
protocol: TCP
port: 8080
targetPort: 5000
nodePort: 30008
Now the fun part: the ingress. And yes, I have an Ingress Controller (nginx) and it works fine.
Ingress:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: test-groq-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
ingressClassName: nginx
rules:
- host: test.com
http:
paths:
- path: /qwen
pathType: Prefix
backend:
service:
name: groq-app-service
port:
number: 8080
- path: /llama
pathType: Prefix
backend:
service:
name: llama-app-service
port:
number: 9090
The problem I need help for: when I try to access one of the two services via its URL, for example http://test.com/qwen, everything is fine, but when I type the input and press Submit, what I get is the URL http://test.com/answer and, obviously, a 404 NOT FOUND error. The URL I'd like to see, with the corresponding web page, is http://test.com/qwen/answer. Obviously, I want something similar for http://test.com/llama.
What does the rewrite, the Ingress or Flask? And how I fix it?
What I tried & I can tell so far:
python or flask commands from both master and worker nodes, while inside a Docker container using docker run, while inside the pod accessing it directly and while inside the pod accessing it through the nodePort. The only problem is the damn URL rewrite.nginx.ingress.kubernetes.io/rewrite-target and regular expressions, but I don't know the right way to do it. I don't even know if it's because of the ingress!I've found the solution: a middleware inside my main.py based on this class. The full code of my main.py is now the following.
from flask import Flask, render_template, request
from groq import Groq
# App configuration
model_id = "qwen/qwen3-32b"
groq_api_key = "<--my API key :)-->"
PREFIX = "/qwen"
# PrefixMiddleware auxiliary class
class PrefixMiddleware:
def __init__(self, app, prefix):
self.app = app
self.prefix = prefix
def __call__(self, environ, start_response):
path = environ.get('PATH_INFO', '')
if path.startswith(self.prefix):
environ['SCRIPT_NAME'] = self.prefix
environ['PATH_INFO'] = path[len(self.prefix):] or '/'
return self.app(environ, start_response)
# App definition
app = Flask(__name__)
app.wsgi_app = PrefixMiddleware(app.wsgi_app, PREFIX)
client = Groq(api_key=groq_api_key)
# Flask routes
@app.route('/')
def index():
return render_template('index.html')
@app.route('/answer', methods=['POST'])
def answer():
input_text = request.form.get('input_text')
if not input_text:
return "Please provide input text.", 400
try:
completion = client.chat.completions.create(
model=model_id,
messages=[
{"role": "system", "content": "User chatbot"},
{"role": "user", "content": input_text}
],
temperature=1,
max_tokens=1024,
top_p=1,
stream=True,
stop=None,
)
result = ""
for chunk in completion:
result += chunk.choices[0].delta.content or ""
except Exception as e:
return f"An error occurred: {e}", 500
return render_template('index.html', input_text=input_text, result=result)
# __main__
if __name__ == "__main__":
app.run(host="0.0.0.0", port=5000)
In the Kubernetes Ingress, I also removed the annotation. The other files are the same.