xmljunitpytestjira-xraypytest-bdd

User-friendly vs. generated name and classname in JUnit XML for import to XRay

Using pytest-bdd with the JUnit argument to produce a nice JUnit XML report.

I then use a cURL command to import the test to XRay, the Atlassian Jira extension.

Jira is running on a server running on our network (not cloud). So I am using something like:

curl --location -X POST "https://jira.nayax.com/rest/raven/1.0/import/execution/junit?projectKey=XXX&testPlanKey=XXX-255" -H "Authorization: Basic Z3VxxxxxxzJMaxxxxxxxxx" -H "Content-Type:multipart/form-data" -F "file=@\"reports/results.xml\""

What I get are tests with names exactly as expected given the XML file. Example fragment:

        <testcase classname="tests.step_defs.test_visitor_page" name="test_create_new_user[110-201]" time="1.125"/>
        <testcase classname="tests.step_defs.test_visitor_page" name="test_create_a_package[110]" time="0.306"/>
        <testcase classname="tests.step_defs.test_visitor_page" name="test_search_user_by_first_name" time="0.700"/>
        <testcase classname="tests.step_defs.test_visitor_page" name="test_scan_nfc[110-10001-TYPE_2]" time="1.399"/>
        <testcase classname="tests.step_defs.test_visitor_page" name="test_get_settings" time="0.669"/>
        <testcase classname="tests.step_defs.test_visitor_page"
                  name="test_get_redemption_machine_for_this_location[110]" time="0.680"/>
        <testcase classname="tests.step_defs.test_visitor_page" name="test_add_free_credits[110]" time="0.766"/>
        <testcase classname="tests.step_defs.test_visitor_page" name="test_buy_credits[110]" time="0.708"/>
        <testcase classname="tests.step_defs.test_visitor_page" name="test_add_free_tickets[110-yes]" time="1.051"/>
        <testcase classname="tests.step_defs.test_visitor_page" name="test_add_free_tickets[110-no]" time="1.034"/>
        <testcase classname="tests.step_defs.test_visitor_page" name="test_purchase_group_package[110]" time="0.968"/>
        <testcase classname="tests.step_defs.test_visitor_page" name="test_get_user_history[110]" time="0.676"/>
        <testcase classname="tests.step_defs.test_visitor_page" name="test_delete_billing_record_from_history[110]"

So you get Jira XRay items named: enter image description here

Is there some decorator or other device I can use to get the Summary fields below (names of the items, like test_buy_credits[110]) to be something like:

Writing an XSD and using an XSLT transformation would probably work, but would be great if there's a simpler way.

Per request, here's are some tests:

@allure.step
@given('Location ID is "<locationId>"', target_fixture="location_id")
def location_id_is_locationid(locationId):
    """Location ID is "<locationId>"."""
    return locationId

@allure.step
@when('I create a package', target_fixture="package_response")
def i_create_a_package(location_id):
    """I create a package"""
    package_response = Package().create(location_id=location_id, session=s)
    return package_response

@then('package is created')
def package_is_created(package_response):
    """Package is created"""
    assert (package_response.status_code == 200)

In answer to the question about my test_ functions, I have to admit that I am pretty new to this, so I only have the given-when-then implementations.

Here is a piece of my feature file:

Feature: Visitor Page
  Scenario Outline: Create new user
    Given Location ID is "<locationId>"
    When I create a new user
    Then call is successful
    And return code is <returnCode>

    Examples: User parameters
      | locationId  | returnCode  |
      | 110         | 201         |

  Scenario Outline: Create a package
    Given Location ID is "<locationId>"
    When I create a package
    Then package is created

    Examples: Package parameters
    | locationId |
    | 110        |

  Scenario Outline: Purchase package
    Given a user exists at "<locationId>"
    And I have created a package
    When user purchases the package
    Then the user will have the package

    Examples:
      | locationId |
      | 110        |

  Scenario: Search user by first name
    Given I have a first name and nfc tag
    When I search for user
    Then I retrieve user details

  Scenario Outline: Scan NFC
    Given a user exists at "<locationId>"
    And box ID is "<boxId>"
    And tag type is "<tagType>"
    When I scan the NFC
    Then nfc tag is scanned

    Examples: NFC parameters
    | locationId | boxId | tagType |
    | 110        | 10001 | TYPE_2  |

  Scenario: Get Settings
    Given I am logged in as an employee
    When I request settings
    Then I get settings
Solution

  • pytest has a Junit XML reporter that uses the name of the test methods for the testcase elements that will be part of the .xml report; AFAIK (and from a quick code inspection), it's not possible to customize these names unless you make a JUnit reporter from scratch. In theory, we could try to get the scenario name from the @scenario mark but I don't an easy way to do so.

    pytest-bdd has a cucumber JSON reporter though that can be more beneficial; Xray supports this format also. To generate the report in Cucumber JSON you would do something as, assuming you have pytest-bdd:

    pytest --cucumber-json cucumber.json

    Please note that the flow related to the processing of Junit XML reports and Cucumber JSON is different though.

    While for ingesting JUnit XML reports, you just need to submit that report to Xray and Xray will do the autoprovision of Test issues, one per each testcase element on the report, the flow for cucumber and other BDD related frameworks is different. You can see here an example (but for Behave and Xray on Jira cloud), to have an idea. There's a bit more detail on a Java related tutorial, detailing the two flows.

    In sum,

    The Xray team doesn't have a tutorial for pytest-bdd (it may be added on the future though), but I was able to get a quick example working.

    This is my feature file features/addition.feature , which contains the tagged feature and scenarios based on the corresponding Story and Test issues respectively.

    @REQ_CALC-1
Feature:  As a user, I can add two numbers

@TEST_CALC-2
Scenario: simple integer addition
        Given I have entered 1 into the calculator
        And I have entered 2 into the calculator
        When I press add
        Then the result should be 3 on the screen

@TEST_CALC-3
Scenario: negative integer addition
        Given I have entered -1 into the calculator
        And I have entered 2 into the calculator
        When I press add
        Then the result should be 1 on the screen

    To run the tests,

    pytest --cucumber-json cucumber.json

    Then, to import these results to a Xray on Jira server/datacenter instance,

    curl -H "Content-Type: application/json" -X POST -u username:password --data @"cucumber.json" https://example.jira.local/rest/raven/2.0/import/execution/cucumber