cucumberbdd

Should I avoid testing on live data for acceptance testing?

My team is transitioning into using BDD and cucumber. We are creating acceptance tests and from what I understand we want to be high level, happy path type tests that are hitting our features end to end as a user would.

How do we:

  1. Ensure the changing data on a live environment won't cause tests to have false failures?
  2. Protect changes deployed to an environment from impacting all customers if things go wrong?

Alternatively we've considered having a sterile, repeatable database that we could use for the app to run against. However, then how would we:

  1. Ensure that we don't miss features failing because the data isn't representative of live data.
  2. Avoid testing too closely to implementation rather than behavior

I would love to get some pointers on best practices here.

Solution

  • Cucumber is a tool and BDD is a process to support the development of new behaviour for a system. As such Cucumber is generally used with a super small set of minimal test data. Its 'tests' (for want of a better word) exercise the behaviour in question and confirm that

    1. Its likely the behaviour is still working
    2. Its likely that the way the behaviour works is still acceptable

    Generally this has nothing to do with how the behaviour works under different live conditions, so your questions about live data are just out of scope.

    Cukes are also generally run on smaller machines (dev boxes, and CI). So again nothing todo with live systems.

    I think you need clear separation between operational stuff and development stuff. An acceptance test is generally considered to be development stuff.

    For example consider the behaviour of being able to login to your system. A cuke would exercise this by having a test database with a single user and that user interacting with the login page to fill in their name and password, submit the form and then be relocated to a page that indicates they are logged in. This would indicate that the code to display the login form, process the login request, authenticate the user etc. etc. works.

    Operationally you might require that with 1 billion users and 250,000 active sessions, you can still login within 200ms.

    The two things are entirely different. The second has nothing to do with BDD and generally is not something you would use Cucumber for (though perhaps you could use Cucumber for part of it)

    Getting clarity between the two very different things would be my first recommendation. Then I suggest you concentrate solely on the single thing that is most important to you. If its operational stuff then BDD in particular and Cucumber are not things you are thinking about. If its developmental, then live environments, data shapes and sizes and deployment are not things you should be thinking about.

    Reducing the number of things you are thinking about at any particular time is good initial step to give you a chance to increase your understanding of something.