cucumbergherkinpython-behavejira-xray

Jira Xray empty Results on Test Details with Behave when import REST execution with Cubumber test type


I'm trying to use Import Execution Results - REST to import Behave JSON results with Multipart. I can get the import to work, but the "The execution details displays the result of the Cucumber Scenario." does not work.

Here's what I'm doing:

I'm using the following feature file with Behave:

Scenario Outline: Verify the UI content of SOME page
    Given I access to SOME in "PAGE" as "DEFAULT_USER"
    When I set "<toggle_type>" toggle "with" page refresh
    Then I can see page displayed with the title "SOME | PAGE"
    And I compare "SOME IMAGE" image with a "98" similarity with "<toggle_type>"

    @SOME-TAG
    Examples:
      | toggle_type |
      | spring      |
      | default     |

I'm running behave with:

behave --no-skipped --tags=SOME-TAG -f json.pretty -o='reports/behave/Xray_behave_result.json'

I'm importing the report with:

POST /rest/raven/1.0/import/execution/behave/multipart

The server reply is:

{"testExecIssue":{"id":"4678506","key":"SOMETHING-55744","self":"https://stage-myjira.somedomain.com/rest/api/2/issue/4678506%22%7D,%22testIssues%22:%7B%22success%22:%5B%7B%22id%22:%224678305%22,%22key%22:%22SOME-TAG%22,%22self%22:%22https://stage-myjira.somedomain.com/rest/api/2/issue/4678305%22%7D%5D%7D,%22infoMessages":\[\]}

But when I look at the Test Details for the test excution:SOMETHING-55744 of the test: SOME-TAG, it's empty: empty results screenshot

This is the resulting json: "Xray_behave_result.json"

[
  {
    "elements": [
      {
        "keyword": "Scenario Outline",
        "location": "something_features/features/something_web.feature:74",
        "name": "Verify the UI content of SOME page -- @2.1 ",
        "status": "passed",
        "steps": [
          {
            "keyword": "Given",
            "location": "something_features/features/something_web.feature:60",
            "match": {
              "arguments": [
                {
                  "name": "page",
                  "value": "some_page"
                },
                {
                  "name": "something",
                  "value": "other thing"
                }
              ],
              "location": "general_common_steps/common_something_steps.py:70"
            },
            "name": "I access to SOME \"page\" as \"DEFAULT USER\"",
            "result": {
              "duration": 11.369250059127808,
              "status": "passed"
            },
            "step_type": "given"
          },
          {
            "keyword": "When",
            "location": "something_features/features/something_web.feature:61",
            "match": {
              "arguments": [
                {
                  "name": "toggle_type",
                  "value": "spring"
                },
                {
                  "name": "refresh_condition",
                  "value": "with"
                }
              ],
              "location": "general_common_steps/common_something_steps.py:362"
            },
            "name": "I set \"spring\" toggle  \"with\" page refresh",
            "result": {
              "duration": 3.7522451877593994,
              "status": "passed"
            },
            "step_type": "when"
          },
          {
            "keyword": "Then",
            "location": "something_features/features/something_web.feature:62",
            "match": {
              "arguments": [
                {
                  "name": "page_title",
                  "value": "SOME | PAGE"
                }
              ],
              "location": "general_common_steps/common_something_steps.py:261"
            },
            "name": "I can see page displayed with the title \"SOME | PAGE\"",
            "result": {
              "duration": 9.731576442718506,
              "status": "passed"
            },
            "step_type": "then"
          },
          {
            "keyword": "And",
            "location": "something_features/features/something_web.feature:63",
            "match": {
              "arguments": [
                {
                  "name": "banner",
                  "value": "SOME IMAGE"
                },
                {
                  "name": "similarity",
                  "value": "98"
                }
              ],
              "location": "general_common_steps/common_something_steps.py:2443"
            },
            "name": "I compare \"SOME IMAGE\" image with a \"98\" similarity with \"spring\"",
            "result": {
              "duration": 0.0009975433349609375,
              "status": "passed"
            },
            "step_type": "then"
          }
        ],
        "tags": [
          "SOME-TAG"
        ],
        "type": "scenario"
      },
      {
        "keyword": "Scenario Outline",
        "location": "something_features/features/something_web.feature:75",
        "name": "Verify the UI content of SOME page -- @2.2 ",
        "status": "passed",
        "steps": [
          {
            "keyword": "Given",
            "location": "something_features/features/something_web.feature:60",
            "match": {
              "arguments": [
                {
                  "name": "page",
                  "value": "SOME"
                },
                {
                  "name": "affiliation",
                  "value": "DEFAULT_USER"
                }
              ],
              "location": "general_common_steps/common_something_steps.py:70"
            },
            "name": "I access to SOME in \"PAGE\" as \"DEFAULT USER\"",
            "result": {
              "duration": 15.28377103805542,
              "status": "passed"
            },
            "step_type": "given"
          },
          {
            "keyword": "When",
            "location": "something_features/features/something_web.feature:61",
            "match": {
              "arguments": [
                {
                  "name": "toggle_type",
                  "value": "default"
                },
                {
                  "name": "refresh_condition",
                  "value": "with"
                }
              ],
              "location": "general_common_steps/common_something_steps.py:362"
            },
            "name": "I set \"default\" toggle \"with\" page refresh",
            "result": {
              "duration": 2.4945812225341797,
              "status": "passed"
            },
            "step_type": "when"
          },
          {
            "keyword": "Then",
            "location": "something_features/features/something_web.feature:62",
            "match": {
              "arguments": [
                {
                  "name": "page_title",
                  "value": "SOME | PAGE"
                }
              ],
              "location": "general_common_steps/common_something_steps.py:261"
            },
            "name": "I can see page displayed with the title \"SOME | PAGE\"",
            "result": {
              "duration": 5.220450162887573,
              "status": "passed"
            },
            "step_type": "then"
          },
          {
            "keyword": "And",
            "location": "something_features/features/something_web.feature:63",
            "match": {
              "arguments": [
                {
                  "name": "banner",
                  "value": "SOME IMAGE"
                },
                {
                  "name": "similarity",
                  "value": "98"
                },
                {
                  "name": "page",
                  "value": "SOME PAGE"
                }
              ],
              "location": "general_common_steps/common_something_steps.py:2443"
            },
            "name": "I compare \"SOME IMAGE\" image with a \"98\" similarity with \"default\"",
            "result": {
              "duration": 0.0029616355895996094,
              "status": "passed"
            },
            "step_type": "then"
          }
        ],
        "tags": [
          "SOME-TAG"
        ],
        "type": "scenario"
      }
    ],
    "keyword": "Feature",
    "location": "something_features/features/something_web.feature:1",
    "name": "SOME NAME",
    "status": "passed",
    "tags": []
  }
  ]

This is the info JSON I am passing:

{
    "fields": {
        "project": {
            "id": "the id of the project"
        },
        "summary": "Brand new Test execution",
        "issuetype": {
            "id": "10007"
        }
    }
}

Solution

  • Behave is not generating a json report compatible with Cucumber. Check the following tutorial that uses a custom reporter to generate a report that, after importation, will show details in each step: https://docs.getxray.app/display/XRAY/Testing+using+Behave+in+Python

    Another possible solution is to use converters that will convert behave reports into compatible Cucumber reports.