[SOLVED] LangChain4j: How to create a service (AIServices) with memory and streaming for responses?

LangChain4j: How to create a service (AIServices) with memory and streaming for responses?

I would like to use a LangChain4j service with memory and streaming.

Code without streaming:

Assistant assistant = AiServices.builder(Assistant.class)
     .chatLanguageModel(model)
     .chatMemory(chatMemory)
     .build();

This code does not work if the type of the model is StreamingChatLanguageModel. How can I do in this case?

Solution

You can declare the return type of the AiService method as a TokenStream:

interface Assistant {

    TokenStream chat(String message);
}

Then create the Assistant:

Assistant assistant = AiServices.builder(Assistant.class)
     .streamingChatLanguageModel(model)
     .chatMemory(chatMemory)
     .build();

Then invoke it with streaming:

TokenStream tokenStream = assistant.chat("Tell me a joke");
tokenStream.onNext(token -> System.out.println(token))
    .onComplete(response -> System.out.println("Streaming completed"))
    .onError(error -> error.printStackTrace())
    .start();

Here is an example, but without a memory: https://github.com/langchain4j/langchain4j-examples/blob/main/other-examples/src/main/java/ServiceWithStreamingExample.java