[SOLVED] Xray robot framework integration issue

To give some context, whenever importing test results through Xray's REST API, usually a Test Execution will be created containing the test results. For each automated test, usually a corresponding Test issue will be created in Xray/Jira, unless an existing one already exists from a previous import. It's also possible to import results to an existing Test Execution issue (i.e., overwrite the results on it) but it's not so frequent.

Therefore, to be able to import test results, in general you need to be able to:

create Test Execution issues on the target Jira project where you're importing the results to
create Test issues on the target Jira project where you're importing the results to

If you are unable to upload test results and obtain a error message on the API call such as { "error": "Error creating Test Execution - Issue create failed!"}, then it can be due to several reasons.

Check that the credentials (client id and client secret) you're using are linked to a user that can create Test Execution and Test issues on the target Jira project. If you're not sure, check with your Jira administrator by going to Xray global settings > API (more info here)

Try to create Test Execution and Test issues on the target Jira project, manually by using Jira's UI. If you have to fill out some additional fields, then it's a sign that your Jira project has been customized and is enforcing those fields to be filled. Therefore, you need to provide those fields whenever submiting results through the REST API. You may have required fields on the Test Execution issue or on the Test issue, or even on both; this will require you to use the "multipart" Xray REST API endpoints (e.g., .../junit/multipart) instead of the "standard" endpoints (e.g., "../junit") to be able to customize values for those custom fields. The "multipart" endpoint syntax of the API request is different from the "standard" one. For each report format, you have a standard a multipart endpoint (JUnit example: standard, multipart).

To be able to set fields on the Test or Test Execution issue, whenever using the multipart endpoint we have to provide a JSON content similar to the one that Jira REST API supports whenever creating issues.´

Example of request, using "curl" utility (usually found in Linux distros, MacOS, and also available for Windows) in the command line:

BASE_URL=https://xray.cloud.getxray.app
token=$(curl -H "Content-Type: application/json" -X POST --data @"cloud_auth.json" "$BASE_URL/api/v2/authenticate"| tr -d '"')
curl -H "Content-Type: multipart/form-data" -X POST -F info=@testExecIssueFields.json -F results=@results.xml -F testInfo=@testIssueFields.json -H "Authorization: Bearer $token" $BASE_URL/api/v2/import/execution/junit/multipart

Next follows an example of one the JSON contents, having fields for the Test Execution issue, setting project key, issue's summary and description, fixVersion, and also setting a custom field with ID "12001", which is a list, to have the value "val_1". There's also another field, with ID 10048, that is a text field so we just have to specify its value.

{
    "fields": {
        "project": {
            "key": "BOOK"
        },
        "summary": "Results for cucumber execution",
        "description": "For more info please check [here|https://www.example.com]",
        "issuetype": {
            "id": "9"
        },
        "customfield_12001" : [
            "val_1"
        ],
        "customfield_10048" : "dummy",
        "fixVersions" : 
                [
                 {
                        "name": "1.0"
                 }
                ]
    }
}

Xray's REST API documentation provides some examples in the multipart endpoints; also, there's a GitHub repository with some source-code snippets that show how to use these endpoints.

Standard fields in Jira (e.g., "summary", "description") can be referred by name in the JSON content that you can use to customize the Test Execution and/or Test issues. Other fields, like the ones created by your project/Jira administrator, need to be referred by their internal ID. You can try to get that ID by yourself, doing a Jira API GET request in your browser to the "field" endpoint ( e.g., https://your_jira_instance/rest/api/3/field). You can also ask your Jira administrator to go to Issues > Fields > Custom Fields and edit the field. The URL will show the ID.

Sometimes the error message returned on the Xray REST API can clearly indicate that there's a specific field that is required.

{
    "error": "Error creating Test Execution - Issue create failed! - assignee: Field 'MyCustomField' cannot be set. It is not on the appropriate screen, or unknown."
}

In this case, this will be due to a Jira configuration issue related to the screen configuration associated with Test Execution issues.

Please check with your Jira administrator that you can manually create Test Execution issues (you can try to create one by yourself) and that you see that field (e.g., "MyCustomField") on the issue creation dialog.

If you don't see it, then use the options on the creation dialog to see if you can add it; if you can't then there's a configuration that needs to be done in Jira's project settings.

Check with your project or Jira admin, under your "Project Settings > Issues > Screens > Issue type screen schemes" configuration which screen scheme is being used and is applicable to your project and the Test Execution and Test issue types. Make sure of the screen that is being used for the "create" and "edit" operations, as sometimes administrators have different screens for these different operations.

Finally, make sure that the field is on the edit and the create screens by editing the related screens.

Additional recommendations if your issue still persists:

try to upload a very simple test result file, without having tests tagged to existing Test issue and without tagging them with requirement/story issue keys
check with your Jira admin that issue types Test Execution and Test are configured for your target project and that there are no special rules and/or customizations enforced in Jira side; as Jira is quite flexible, sometimes Jira administrators can customize many of its behaviours, including using third-party apps to implement further customizations & control