chatbotrasa

Botium CLI tests for rasa chatbot passing for any random convo (that should not pass)


I'm trying to setup Botium CLI with my rasa chatbot for automated integration testing and dialog flow tests. However, the botium framework passes tests that do not describe a conversation flow that would be possible with my chatbot.

I'm using it with botium-connector-rasa and this is my botium.json config file:

{
  "botium": {
    "Capabilities": {
      "PROJECTNAME": "change me later",
      "CONTAINERMODE": "rasa",
      "RASA_MODE": "DIALOG_AND_NLU",
      "RASA_ENDPOINT_URL": "http://localhost:8080/"
    },
    "Sources": {},
    "Envs": {}
  }
}

When I try to run botium-cli pointing --convos to my folder with the .convos.txt files, it passes the tests even if they should have failed.

.convo.txt file:

Test case 02: Robots' hell

# me
random question

# bot
random answer

Command used for running the tests:

botium-cli run --config botium.json --convos ./convos/

The output is this

What is going on? Why is botium passing my random tests when it should've failed these tests?

I've tried to talk with the bot using the emulator and if i run botium-cli emulator it works properly and I can communicate with my chatbot as expected.


Solution

  • The issue was in the .convo.txt files' syntax. I just had to remove the spaces between the # and the me/bot. The provided example convo should look like this instead:

    Test case 02: Robots' hell
    
    #me
    random question
    
    #bot
    random answer