[SOLVED] StreamingTextResponse doesn't working correctly in production

StreamingTextResponse doesn't working correctly in production

Code which I shared - is api route in Next.js In dev mode, all working as I expected, but in production, the response seems like static response, so data is sending not dynamically, just like one part

I'm not sure why that happens

On the client side it handled by TextDecoder class, I get chunks with text from api route and add them to state

So as I said, in the dev mode all working like readable stream response, but in production not

used libraries langchain and ai

import { ChatOpenAI, ChatOpenAICallOptions } from "langchain/chat_models/openai"
import { PromptTemplate } from "langchain/prompts"
import { BytesOutputParser } from "langchain/schema/output_parser"
import { StreamingTextResponse } from "ai"
import { cookies } from "next/headers"
import { NextRequest, NextResponse } from "next/server"

export async function POST(req: NextRequest): Promise<StreamingTextResponse> {

    if (!cookies().has(/*property_name*/)) {
        return NextResponse.json(null, { status: 401, statusText: "Unauthorized" })
    }
    // Request body
    const body: AssistantPayload = await req.json()
    // Messages variable
    let messages: Message[] = body.messages
    // Initialize chat model instance
    const model: ChatOpenAI<ChatOpenAICallOptions> = new ChatOpenAI({
    // ... config properties
    })
    // Parser instance
    const outputParser = new BytesOutputParser()

    const prompt = PromptTemplate
        .fromTemplate(/* I can't share it. It's just system prompt */)

    const history = JSON.stringify( // Create history JSON
        messages.map(m => /* creating required history structure as string */)
    const content = body.prompt // New user message
    const chain = prompt.pipe(model).pipe(outputParser)

    const stream = await chain.stream({ history, content }) // Create readable stream

    return new StreamingTextResponse(stream) // Readable stream response
}

Solution

I was able to make the streaming works for production builds adding:

export const dynamic = 'force-dynamic';

at the beginning of API route that is implementing the streaming.

Source:

https://github.com/vercel/next.js/discussions/48427#discussioncomment-5624604