seleniumspecflowazure-pipelinesazure-pipelines-build-tasknunittestadapter

VSTS is it possible to run selenium UI test tasks against localhost web app?

I might be missing something obvious here but my mvc web app is successfully tested using specflow UI tests on my dev machine using localhost but I can't get this to work on a CI Build pipeline in VSTS?

Here is my setup, 1 x Asp.Net solution comprising of:

On my Dev machine: I can run the unit tests using NUnit3TestAdapter within visual studio, with my website running at localhost:58909

On VSTS: After the build task, the Visual Studio Test Task is picking up the NUnit Adapter and tries to run the tests (using InternetExplorerDriver server) but fails to find my website running on localhost it seems:

Here's the output:

2017-04-03T09:02:31 Starting test execution, please wait...
2017-04-03T09:02:32 Information: NUnit Adapter 3.7.0.0: Test execution started
2017-04-03T09 Information: Running all tests in C:\a\1\s\WebCalculator.UiTestsNUnitRunner\bin\Release\WebCalculator.UiTestsNUnitRunner.dll
2017-04-03T09:02:32 Information: NUnit3TestExecutor converted 3 of 3 NUnit test cases
2017-04-03T09:02:33  Started InternetExplorerDriver server (32-bit)
2017-04-03T09:02:33  3.0.0.0
2017-04-03T09:02:33  Listening on port 1065
2017-04-03T09:02:33  Only local connections are allowed
2017-04-03T09:03:54  Started InternetExplorerDriver server (32-bit)
2017-04-03T09:03:54  3.0.0.0
2017-04-03T09:03:54  Listening on port 1123
2017-04-03T09:03:54  Only local connections are allowed
2017-04-03T09:03:54  Failed   AddTwoNumbers("50","70","120",System.String[])
2017-04-03T09:03:54  ##[error]Error Message:
2017-04-03T09:03:54  ##[error]   OpenQA.Selenium.WebDriverTimeoutException : Timed out after 10 seconds
2017-04-03T09:03:54  ##[error]  ----> OpenQA.Selenium.NoSuchElementException : Unable to find element with id == summandOne

So my question, is the Visual Studio Test Task smart enough to test against my website which is built as part of the solution at localhost:58909 ?

The example from MS here: https://www.visualstudio.com/en-us/docs/test/continuous-testing/getting-started/continuous-test-selenium

Has the Specflow tests pointing at an XXX.azurewebsites.net url so I'm starting to think that building the website and executing the UI tests together (like I'm attempting) isn't the right approach for this?

Is it the case that when you want to run Specflow tests against a website - I need to build and release this to a hosted environment first??

Solution

  • Based on the source and test paths (c:\a\1\s…), I assume you are using Hosted Agent and do test on build agent. You can’t do it.

    First, the Hosted agent is running as service, it can’t run interactive test (selenium UI test).

    Secondly, the localhost means the test machine and website machine are at the same machine. Obviously, your website isn’t running in Hosted Agent.

    Regarding the article you mentioned, the website is in the azure (baseURL=”https://partsumlimited.azurewebsite....), so it can be access from internet

    You can refer to these steps to achieve your requirement:

    1. Setup a on premise build agent that running as interactive mode (if it is running as service mode, you need to do test on remote machine, refer to that article to deploy agent, copy file and run test) Deploy an agent on Windows
    2. Create a new build definition to use that build agent
    3. Add IIS Web App Deployment step or other to deploy your web app to IIS (Deploy your Web Deploy package to IIS servers using WinRM)
    4. Change your website URL in test code (You can use localhost if the build agent is running on the same machine)

    Note: If you want to run test on remote machine, the remote machine need to be accessible from build agent.